RexBERT: Context Specialized Bidirectional Encoders for E-commerce

AI summary

Overview

• Research area: Natural Language Processing — domain-specific (e-commerce) encoder-only transformer language models, pre-training data curation, and evaluation on retrieval/classification/similarity benchmarks.
• Technical level: Intermediate. It helps to know what BERT-style masked language modeling, GLUE, and embedding-based retrieval are, but the paper's arguments are laid out in accessible terms.
• Scope (1 sentence): The paper releases a 350 billion token e-commerce corpus, a three-phase pre-training recipe built on ModernBERT's architecture, and a family of RexBERT encoders from 17M to 400M parameters that outperform larger general-purpose encoders on e-commerce tasks.

What This Paper Is About

Most encoder-only transformers are trained on generic web text and therefore miss the specific vocabulary and structure of shopping content — product titles, attribute lists, and the subtle difference between a substitute product and a complementary one. The authors set out to build encoders specialized for e-commerce using only open data, testing whether careful in-domain data curation and a staged training curriculum can beat simply making models bigger. The goal is a family of efficient encoders that improve search, ranking, and catalog tasks without sacrificing general language ability.

Key Contributions

1. Ecom-niverse corpus. A 350 billion token collection of retail-relevant text distilled from the 4.4 trillion token CommonCrawl-derived FineFineWeb corpus, covering categories such as fashion, beauty, automotive and entertainment, produced by a modular filtering pipeline (domain selection, sampling, LLM labeling with Phi-4, QA auditing with Llama3-70B, fastText distillation, and thresholded filtering at scale).
2. A reproducible three-phase pre-training recipe built on ModernBERT's architectural advances: general pre-training, context extension to 8,192 tokens, and annealed domain specialization that incorporates a Guided MLM masking strategy.
3. A family of RexBERT models at four scales — 17M, 68M, 150M and 400M parameters (Micro, Mini, Base, Large) — with configurations documented in the paper.
4. Evaluation across token classification, semantic similarity, and general NLU, using e-commerce benchmarks derived from the Amazon ESCI dataset plus the GLUE benchmark, showing that smaller in-domain models can beat larger general-purpose ones.

Main Findings

• Token recovery gains over ModernBERT-base: On product titles, RexBERT-base improves top-1 accuracy from 60.5% to 69.2% at a 128-token block size and from 65.5% to 72.6% at 512 tokens. On product descriptions, it improves top-1 accuracy from 67.8% to 73.1% at 128 tokens and from 74.0% to 77.4% at 512 tokens.
• A small model beats a larger baseline: The 68M-parameter RexBERT-mini surpasses ModernBERT-base on product titles at 128 tokens, with top-1 accuracy of 65.58% versus 60.5% — a +5.08 point gain.
• Large-scale token classification: RexBERT-large reaches 72.0 / 82.3 / 85.1 top-1/3/5 accuracy on product titles at 128 tokens, rising to 76.1 / 85.6 / 88.2 at 512 tokens; on descriptions it reaches 75.7 / 86.8 / 89.8 at 128 tokens and 79.7 / 89.6 / 92.1 at 512 tokens.
• Semantic similarity: RexBERT consistently yields higher Spearman correlation than general-purpose encoders at comparable scale, with the RexBERT series (Micro to Large) exceeding ModernBERT, Ettin, and EmbeddingGemma in Figure 4. Gains are strongest going from Micro to Mini to Base, while returns are described as "decent" at the Large scale. Convergence was stable across random seeds. The paper does not enumerate the numeric correlation values in the text; they are shown in Figure 4.
• General NLU transfers: Despite being specialized for e-commerce, RexBERT-large leads on paraphrase detection (QQP 89.84 vs. 89.73), multi-genre inference (MNLI 92.06 vs. 90.14) and semantic similarity (STS-B 91.80 vs. 91.33), while ModernBERT-large leads on SST-2 (96.67 vs. 95.76) and QNLI (94.82 vs. 94.00).
• Small models are consistently stronger in-domain: RexBERT-mini outperforms DistilBERT across all five GLUE tasks, and RexBERT-micro outperforms ettin-17m on all evaluated tasks (for example SST-2 90.37 vs. 88.42; QNLI 85.59 vs. 85.30).
• Data and curriculum beat raw scale: The authors conclude that on strongly e-commerce-shaped distributions, representation quality is bottlenecked less by model capacity than by exposure to the right long-tail entities, attribute-value constructions, and overall training token set.

Methodology in Plain English

The authors start from raw web data and carve out the shopping-relevant parts. They pick FineFineWeb categories likely to overlap with commerce (split into those needing filtering, like Hobby, Health, and Travel, and those selected directly, like News, Entertainment, and Fashion), sample hundreds of thousands of items per domain, and have an instruction-tuned model (Phi-4) label each item as Relevant or Not Relevant. A stronger model (Llama3-70B) audits a subset of those labels in a QA loop until consistency stabilizes. Those LLM labels are then distilled into lightweight per-domain fastText classifiers, validated on held-out data, and run at scale with a per-domain score threshold to produce the final corpus. Language detection retains English, and MinHash deduplication plus profanity and adult-content filtering clean the result.

Training follows a three-stage curriculum on a ModernBERT-style encoder. Phase 1 trains on a 1.7 trillion token mixture of web text, books, code, technical papers and multilingual content at a short 512-token sequence length with a 30% masking ratio. Phase 2 extends the maximum sequence length to 8,192 tokens over 250 billion tokens, switching to rotary positional embeddings with NTK-aware scaling and alternating global and local attention. Phase 3 anneals the sampling weights toward Ecom-niverse for roughly 350 billion tokens, lowering the masking ratio to 10%–15%. In this final phase the authors add Guided MLM, which uses a lightweight entity and attribute mining pipeline to find domain-relevant spans and preferentially mask them; about 5% of sequences per batch come from this guided distribution and the rest from standard span masking. The tokenizer is a modern BPE derived from OLMo with a vocabulary of 50,368 tokens, next-sentence prediction is dropped, and optimization uses StableAdamW with a trapezoidal Warmup–Stable–Decay learning rate schedule.

Evaluation uses the Amazon ESCI (Shopping Queries) dataset — roughly 130,000 unique search queries and 2.6 million human-labeled query-product pairs across English, Spanish and Japanese, with four relevance levels (Exact, Substitute, Complement, Irrelevant) — restricted to the English subset. Two tasks are tested: masked token recovery on product titles and descriptions truncated to 128, 256 or 512 tokens with 15% of tokens masked, scored at top-1, top-3 and top-5; and semantic similarity, where the four labels are mapped to 1.0, 0.66, 0.33 and 0.0 and models are fine-tuned with CoSENT loss and scored by Spearman correlation. GLUE provides the general-language control.

Why This Matters

This paper is a case study in arguing that domain-specialized encoders trained on carefully curated open data can outperform indiscriminate scaling. It also matters as a reproducibility contribution: the corpus pipeline, the three-phase recipe, and four model sizes are all described in enough detail to be repeated or transplanted to other verticals, and the authors explicitly frame the approach as domain-agnostic (healthcare, legal, scientific, and others).

Real-world applications:

• Catalog quality and attribute completion: stronger masked-token recovery on titles and descriptions supports better attribute extraction and normalization.
• Search and candidate generation: higher Spearman correlation on graded relevance implies more faithful embedding geometry, especially for distinguishing substitutes from complements, which generic models often conflate.
• Long merchant pages and multi-section descriptions: a single encoder handling 8,192 tokens reduces reliance on truncation heuristics that may drop critical attributes.
• Compliance and routing/classification: encoder-only models remain the production workhorses where latency, stability and cost matter.

Industry relevance: e-commerce platforms care about inference cost, so a model that matches or beats larger baselines with 2–3x fewer parameters is directly attractive. The result also gives practitioners a template for building specialized encoders from open corpora rather than licensing proprietary data.

Future Directions

1. Isolating Guided MLM. The paper states explicitly that it does not attempt an ablation to separate Guided MLM's contribution from corpus composition and annealing, and calls for controlled studies to quantify its marginal effect.
2. Scaling behavior at the top end. Semantic similarity gains were strongest from Micro to Mini to Base but described as decent at the Large scale, raising the question of where the returns from in-domain data genuinely stop.
3. Porting the recipe to other verticals. The authors propose replacing Ecom-niverse with a well-curated corpus from another domain and applying the same pipeline, but do not demonstrate this transfer themselves.
4. Non-English and multilingual e-commerce. ESCI includes Spanish and Japanese, but the authors restricted experiments to the English subset, leaving multilingual specialization untested.

Target Audience

Machine learning engineers and applied NLP researchers building search, ranking or classification systems for e-commerce; practitioners interested in domain-adaptive pre-training and data curation pipelines; and researchers studying the trade-off between model scale and data quality. Readers focused on decoder-only generative models will find less of direct interest, since the paper is exclusively about encoders.

Authors’ abstract

Encoder-only transformers remain indispensable in retrieval, classification, and ranking systems where latency, stability, and cost are paramount. Most general purpose encoders, however, are trained on generic corpora with limited coverage of specialized domains. We introduce RexBERT, a family of BERT-style encoders designed specifically for e-commerce semantics. We make three contributions. First, we release Ecom-niverse, a 350 billion token corpus curated from diverse retail and shopping sources. We describe a modular pipeline that isolates and extracts e-commerce content from FineFineWeb and other open web resources, and characterize the resulting domain distribution. Second, we present a reproducible pretraining recipe building on ModernBERT's architectural advances. The recipe consists of three phases: general pre-training, context extension, and annealed domain specialization. Third, we train RexBERT models ranging from 17M to 400M parameters and evaluate them on token classification, semantic similarity, and general natural language understanding tasks using e-commerce datasets. Despite having 2-3x fewer parameters, RexBERT outperforms larger general-purpose encoders and matches or surpasses modern long-context models on domain-specific benchmarks. Our results demonstrate that high quality in-domain data combined with a principled training approach provides a stronger foundation for e-commerce applications than indiscriminate scaling alone.