WebFAQ 2.0: A Multilingual QA Dataset with Mined Hard Negatives for Dense Retrieval

AI summary

Overview

Research area: Information retrieval (cs.IR) — multilingual and cross-lingual dense retrieval, dataset construction, and hard negative mining.

Technical level: Intermediate. The dataset construction is easy to follow, but the training objectives (MultipleNegativesRankingLoss, MarginMSE) and the retrieval evaluation setup assume some familiarity with dense retrieval.

Scope: The paper releases and describes WebFAQ 2.0, a large multilingual FAQ-derived question-answer corpus, plus a companion hard negatives dataset with cross-encoder scores, and evaluates how training dense retrievers on those negatives compares with random-negative training.

What This Paper Is About

Building good multilingual search and question-answering systems requires large, diverse, multilingual training data, and such resources are scarce. The authors rebuild and greatly expand the WebFAQ dataset — question-answer pairs extracted from schema.org FAQ markup on websites — by crawling the web directly rather than relying on preprocessed dumps, and they add a new resource of "hard negatives" (plausible but wrong answers) for training dense retrievers. The goal is to give the research community a bigger, more multilingual QA corpus along with the negatives needed to train and stress-test retrieval models, while honestly reporting where those hard negatives help and where they hurt.

Key Contributions

1. A much larger and more multilingual QA dataset. WebFAQ 2.0 contains approximately 198 million QA pairs, described in the abstract as spanning 108 languages and in the introduction as 104 languages, compared with 96 million QA pairs in 75 languages in the original release.

2. A new collection strategy. Instead of relying solely on Web Data Commons structured data dumps, the authors mined URLs containing the term FAQPage from Common Crawl dumps from 2025 and crawled the pages directly with the OWLer distributed crawler, which also captures hreflang links to multilingual versions of the same page and additional metadata such as page titles and descriptions.

3. Expanded bilingual QA alignments. Using LaBSE embeddings with a minimum similarity threshold of 0.9, the dataset now contains over 14.3M aligned QA pairs across 3,970 language combinations, up from 1.5M aligned pairs across 1,001 language pairs in v1 (the introduction describes the cross-lingual increase as going from 1.5 million to over 13.8 million).

4. A mined hard negatives dataset with cross-encoder scores. The release contains exactly 1.25 million quintuples of the form (query, positive, pos_score, negatives, neg_scores) across 20 languages, with 200 cross-encoder-scored negatives per query, enabling both Contrastive Learning with MultipleNegativesRankingLoss and Knowledge Distillation with MarginMSE.

Main Findings

• English share dropped sharply. English fell from over 50% of the data (49M pairs) to under 30% (55M pairs), reflecting deliberate multilingual expansion. Per-language shares in Table 1 (v1 in parentheses): English 27.9 (51.2), Spanish 8.8 (6.0), German 7.3 (6.9), French 6.5 (4.8), Russian 4.4 (3.8), Dutch 4.3 (2.8), Portuguese 4.0 (1.7), Italian 3.5 (2.7), Polish 2.5 (1.7), Other 30.8 (13.0). Individual languages grew sharply: Hindi from 537k to 2.6M and Ukrainian from 806k to 2.6M.

• Topic distribution shifted heavily toward travel. Traveling and Hospitality rose to 59.2% of the dataset (34.1 in v1), while several categories shrank: Healthcare Services, Wellness and Lifestyle 5.6 (13.0), Entertainment, Recreation, and Leisure 4.7 (9.7), Employment, Education, and Training 3.5 (9.5), Banking, Financial Services, and Insurance 4.4 (6.0), Legal Services, Regulations and Government 2.3 (4.0), Products and Commercial Services 18.9 (19.8), General Information and Other 4.4 (3.9). Topic labels came from a fine-tuned XLM-RoBERTa classifier trained for five epochs on 79,650 GPT-5-mini annotated samples drawn from the 49 most common languages, with an 80/10/10 split and 88.07% F1 on the test set.

• Most "questions" are not actually questions. The question-type distribution shows Not-a-Question at 67,870,478 (34.7%), then Factoid 65,601,236 (33.6%), Experience 22,649,722 (11.6%), Instruction 20,179,307 (10.3%), Evidence-Based 8,300,537 (4.3%), Reason 6,503,638 (3.3%), Comparison 4,236,223 (2.2%), and Debate 107,626 (0.1%). These labels came from an XLM-RoBERTa model trained on roughly 48,300 labeled instances across 49 languages for 6 epochs, with a learning rate of 2×10⁻⁵ and batch size 16, reaching about 88.1% F1. The labels were generated by an ensemble of LLaMA 3.1, Gemma 2, and Qwen 2.5 using majority vote, and the authors describe this classification step as preliminary and needing further refinement.

• Bitext alignment grew most strongly between non-English pairs. The largest alignment count is Marathi–Telugu, which went from 0 in v1 to 89,910 in v2. Other top pairs: German–Spanish 19,739 to 80,623; Russian–Ukrainian 15,251 to 74,545; Italian–Portuguese 10,924 to 69,469; Indonesian–Korean 1,515 to 42,986. Restricting to the 1,282 language pairs with at least 4,000 aligned samples still leaves more than 9.6M aligned QA pairs. Alignment quality was validated with the GEMBA metric based on LLM judgments, and a new bitext mining task was contributed to the MTEB benchmark.

• Random negatives beat hard negatives in contrastive training. False negatives are prevalent in the mined hard negatives, so models trained with random negatives (MNR RN) outperform unfiltered hard negatives overall. On WebFAQ Retrieval, NDCG@10 for MNR (RN) was ara 74.3, eng 60.0, ind 81.9, jpn 69.8, kor 81.3, rus 62.0, versus MNR Top4 at 71.8, 48.7, 78.8, 63.6, 81.4, 51.4. The gap is more pronounced for Top4 than for the denoised variant (74.8, 57.8, 82.2, 69.6, 83.1, 59.9).

• Knowledge distillation gives the best non-English results but costs English performance. MarginMSE (M-MSE) outperformed the MNR configurations across most non-English languages. On WebFAQ Retrieval, M-MSE scored ara 75.4, eng 57.4, ind 78.5, jpn 69.7, kor 81.5, rus 61.1. The authors attribute the English trade-off to the base model being in-domain pretrained on MS MARCO (exclusively English) and to the large English share in the hard negatives.

• M-MSE was strongest on the other two benchmarks as well. On MIRACL (Hard Negatives), M-MSE reached ara 54.7, eng 34.7, ind 38.7, jpn 46.0, kor 50.9, rus 42.6, compared with MNR (RN) at 50.5, 37.2, 36.1, 39.8, 42.4, 39.6 and Base at 36.1, 30.6, 28.1, 27.6, 39.3, 29.6. On Mr. TyDi, M-MSE reached ara 50.4, eng 27.7, ind 51.4, jpn 35.6, kor 37.6, rus 40.3, versus MNR (RN) at 44.7, 29.5, 47.7, 33.5, 34.7, 30.8 and Base at 30.5, 18.1, 33.2, 16.0, 27.7, 22.5. BM25 is reported as a reference for all three benchmark sets.

• The resource is continuously updated. Since late 2025 structured FAQs are regularly released through the Open Web Index, with new dumps published daily, and the introduction dates this to November 2025.

Methodology in Plain English

Dataset construction. The authors took the URL list from the first WebFAQ and added more URLs found by scanning Common Crawl 2025 raw HTML for the term FAQPage, which marks schema.org FAQ markup. They downloaded those pages with the OWLer crawler. Questions and answers were parsed from the markup, HTML tags cleaned, malformed entries removed, and languages detected with FastText. A key change from v1: the old heuristic for removing near-duplicate questions (using Jina v3 embeddings) was dropped because choosing a similarity threshold is difficult; instead, retrieval test sets take only one QA pair per domain so duplicates cannot distort evaluation. Jina v3 embeddings are still used to compute and release a question-answer semantic similarity score, which helps flag ambiguous pairs — the paper's example being "Will it be painful?" whose meaning only becomes clear from the page title "Doubleview Podiatry – Walk with Confidence."

Bilingual alignments. All questions and answers across language pairs are embedded with LaBSE, nearest neighbors are retrieved to form aligned pairs, and only pairs above a LaBSE similarity of 0.9 are kept.

Hard negative mining. For each query, BM25 retrieves the top 200 candidate answers from the matching monolingual WebFAQ corpus; these are reranked with the BGE-m3 cross-encoder. The output quintuple includes the query, the positive answer, the positive score, the negatives, and their scores. The cross-encoder scores are intended to support denoising in the style of RocketQA, filtering out candidates by an adaptive threshold to reduce false negatives.

Training experiments. The base model is XLM-RoBERTa base, in-domain pretrained on MS MARCO with MarginMSE loss. All models are fine-tuned on the 1.25 million training samples across 20 languages for one epoch with batch size 128 and a negative sampling ratio of 1:4. Three configurations are compared: random negatives; hard negatives with MNR using Top4 or Denoised selection (Denoised discards negatives with cross-encoder scores above 0.9 or below 0.1 and randomly samples 4 from the rest); and M-MSE using the top 10 ranked hard negatives per query as soft targets. Evaluation uses NDCG@10 on WebFAQ Retrieval (in-domain, 20 languages), MIRACL-HN, and Mr. TyDi, reported for Arabic, English, Indonesian, Japanese, Korean, and Russian.

Why This Matters

Impact on research. The paper provides a substantially larger and more linguistically balanced QA corpus than prior FAQ-based resources such as CCQA, PAQ, and WebFAQ v1, plus a reusable hard negatives resource with cross-encoder scores. Its honest reporting of false negatives and the English/non-English trade-off in distillation gives the community concrete guidance rather than a uniformly positive result.

Real-world applications:

• Multilingual customer support and help-desk search, where FAQ content is the dominant data source.
• Cross-lingual search over websites that publish the same FAQ in multiple languages, supported by the bilingual alignments.
• Training and evaluating embedding models for low-resource and non-European language pairs (for example Marathi–Telugu, Russian–Ukrainian, Indonesian–Korean).
• Benchmarking dense retrieval robustness under hard negative pressure, via the WebFAQ Retrieval MTEB task, the new bitext mining MTEB task, and evaluation on MIRACL-HN and Mr. TyDi.

Industry relevance. Companies building multilingual search, RAG, or semantic embedding products need exactly this combination: in-domain training pairs, hard negatives, and student-teacher distillation targets. The finding that random negatives remain competitive in contrastive setups is directly relevant to teams allocating compute for training data mining.

Future Directions

1. Reducing false negatives. False negatives remain the central obstacle in the mined hard negatives; better filtering, LLM-based relevance judging, or variance-based selection (as in NV-Retriever) could make hard negatives reliably outperform random negatives.

2. Resolving the English performance trade-off. Gains from MarginMSE in non-English languages came at the cost of English retrieval; how to preserve English quality while distilling multilingual cross-encoder scores is left open.

3. Refining the question-type and topic labels. The question-type classifier is explicitly described as preliminary because its labels came from LLM ensembles rather than human annotation, and it needs further refinement.

4. Growing the dataset continuously. The authors plan to exploit the Open Web Index, where structured data including FAQs has been published regularly since late 2025 with daily dumps, to build more extensive and timely multilingual QA benchmarks.

Target Audience

Researchers and engineers working on multilingual and cross-lingual information retrieval, dense retriever training, and text embedding models; dataset builders interested in web-scale QA extraction and bitext mining; and practitioners who need multilingual FAQ-style training or evaluation data for search and question-answering systems. Readers should be comfortable with NDCG@10, contrastive training losses, and cross-encoder reranking to get the most from the evaluation section.

Authors’ abstract

We introduce WebFAQ 2.0, a new version of the WebFAQ dataset, containing 198 million FAQ-based natural question-answer pairs across 108 languages. Compared to the previous version, it significantly expands multilingual coverage and the number of bilingual aligned QA pairs to over 14.3M, making it the largest FAQ-based resource. Unlike the original release, WebFAQ 2.0 uses a novel data collection strategy that directly crawls and extracts relevant web content, resulting in a substantially more diverse and multilingual dataset with richer context through page titles and descriptions. In response to community feedback, we also release a hard negatives dataset for training dense retrievers, with 1.25M queries across 20 languages. These hard negatives were mined using a two-stage retrieval pipeline and include cross-encoder scores for 200 negatives per query. We further show how this resource enables two primary fine-tuning strategies for dense retrievers: Contrastive Learning with MultipleNegativesRanking loss, and Knowledge Distillation with MarginMSE loss. WebFAQ 2.0 is not a static resource but part of a long-term effort. Since late 2025, structured FAQs are being regularly released through the Open Web Index, enabling continuous expansion and refinement. We publish the datasets and training scripts to facilitate further research in multilingual and cross-lingual IR. The dataset itself and all related resources are publicly available on GitHub and HuggingFace.