TurkColBERT: A Benchmark of Dense and Late-Interaction Models for Turkish Information Retrieval

AI summary

Overview

• Research area: Neural information retrieval (IR) for Turkish, a morphologically rich and comparatively lower-resource language; specifically the comparison of dense bi-encoder retrieval against late-interaction (ColBERT-style) retrieval, plus multi-vector indexing efficiency.
• Technical level: Intermediate. The paper assumes familiarity with dense retrieval, token-level interaction, and approximate nearest neighbor search, but explains its pipeline step by step.
• Scope in one sentence: TurkColBERT is presented as the first comprehensive benchmark comparing dense encoders and late-interaction models for Turkish retrieval, evaluating 10 models across five Turkish BEIR datasets and comparing three indexing strategies (PLAID, MUVERA, and MUVERA with reranking).

What This Paper Is About

Neural retrieval systems work well in high-resource languages, but they remain underexplored for Turkish, whose agglutinative morphology makes fixed single-vector document representations lossy. Dense bi-encoders currently dominate Turkish IR, while late-interaction models (which keep token-level representations for fine-grained matching) have not been systematically evaluated for Turkish.

The authors build a reproducible training and evaluation pipeline that adapts English and multilingual encoders to Turkish and then converts them into ColBERT-style retrievers, so that dense and late-interaction models can be compared under identical conditions. They also test whether efficient multi-vector indexing algorithms make late-interaction retrieval practical for production use in Turkish.

Key Contributions

1. TurkColBERT, a Turkish IR benchmark. The paper describes it as the first comprehensive benchmark comparing dense encoders and late-interaction models for Turkish retrieval. It evaluates 10 models across five Turkish BEIR collections: SciFact-TR, Arguana-TR, Fiqa-TR, Scidocs-TR, and NFCorpus-TR, covering scientific, financial, argumentative, and nutrition domains.
2. A two-stage adaptation pipeline. English and multilingual encoders are first fine-tuned on Turkish NLI and STS tasks (all-nli-tr and STSb-TR), then converted into ColBERT-style retrievers with PyLate trained on MS MARCO-TR. A third stage integrates MUVERA multi-vector indexing.
3. A parameter-efficiency result. The 1.0M-parameter colbert-hash-nano-tr is reported as 600× smaller than the 600M-parameter turkish-e5-large dense encoder while preserving over 71% of its average mAP. Late-interaction models that are 3–5× smaller than dense encoders are reported to significantly outperform them.
4. An indexing-algorithm comparison for production readiness. MUVERA+Rerank is reported as 3.33× faster than PLAID with a +1.7% relative mAP gain, and ColmmBERT-base-TR is reported to reach 0.54 ms query times under MUVERA.
5. Released artifacts. All checkpoints, configs, and evaluation scripts are released to support reproducibility.

Main Findings

• Late interaction beats dense on most Turkish benchmarks. ColmmBERT-base-TR (310M parameters) reaches the highest mAP on four of the five benchmarks: SciFact-TR (56.8%), Fiqa-TR (19.5%), Scidocs-TR (6.8%), and NFCorpus-TR (11.5%). On SciFact-TR it reaches 70.0% Recall@10, beating TurkEmbed4Retrieval (60.5%) and turkish-e5-large (63.3%) by 6.5 to 9.5 percentage points. The abstract states ColmmBERT-base-TR yields up to +13.8% mAP on domain-specific tasks, and the conclusion describes up to 87% improvement in mAP on domain-specific tasks such as financial QA.
• Dense models still win on argument retrieval. turkish-e5-large secures the top mAP on Arguana-TR (17.9%), against ColmmBERT-base-TR's 17.3%, which the authors attribute to the need for semantic breadth in argument-focused tasks.
• Dense models struggle in specialized domains. On Scidocs-TR, turkish-e5-large achieves only 2.2% mAP while ColmmBERT-base-TR reaches 6.8%, which the paper describes as a 209% improvement.
• Small late-interaction models are competitive. ColmmBERT-small-TR (140M parameters) achieves 70.3% Recall@10 and 55.4% mAP on SciFact-TR, described as about 97.5% of the full 310M-parameter model's performance while requiring less than half the computational resources. The conclusion states it operates at only 45% of the computational cost.
• Ultra-compact models trade quality for size. colbert-hash-nano-tr (1.0M parameters, described as 310 times smaller than the base model) retains 63.7% of base mAP on SciFact-TR. The smallest variant, colbert-femto-tr (0.2M parameters), reaches 19.0% mAP, which the authors describe as below production viability.
• Dataset difficulty varies enormously. SciFact-TR is the easiest dataset, with multiple models exceeding 70% R@10, while Scidocs-TR is the most demanding, peaking at just 10.4%. The paper reports disparities often exceeding 500% across metrics.
• Indexing configuration strongly affects latency. Reported query latency ranges from an ultra-fast 0.72 ms with plain MUVERA, up to 73–124 ms with PLAID-based approaches, with MUVERA+Rerank in between at 27–35 ms. With TurkEmbed4Retrieval, MUVERA+Rerank yields 0.5253 NDCG@100 versus PLAID's 0.3257, while cutting latency from 73.6 ms to 35.2 ms. The conclusion states MUVERA+Rerank maintains 90–95% retrieval quality.
• Semantic fine-tuning improves base encoders. After the two-stage methodology, mmBERT-small reaches a Spearman correlation of 0.78 on STSb-TR and 93% triplet accuracy on AllNLI-TR, outperforming the pretrained baseline by +22% and +26% respectively.
• Table coverage note. The model overview table (Table 2) lists 10 models ranging from 0.2M to 600M parameters, while the results table (Table 3) additionally reports two entries not listed in that overview: mxbai-edge-colbert-v0-17m-tr and mxbai-edge-colbert-v0-32m-tr.

Methodology in Plain English

Stage 1 — Learning Turkish sentence meaning. The team started from publicly available checkpoints of several model families: mmBERT (base and small), Ettin encoders, and BERT-Hash variants (nano, pico, femto). They used the Sentence Transformers framework with mean pooling over the final layer to produce sentence embeddings, then fine-tuned first on all-nli-tr (Turkish translations of SNLI and MultiNLI formatted as anchor–positive–negative triplets) and then on STSb-TR (sentence pairs with continuous similarity scores from 0 to 5).

The NLI training used MultipleNegativesRankingLoss wrapped in MatryoshkaLoss so that embeddings work at multiple dimensions, [768, 512, 384, 256, 128, 64] for base models and [384, 256, 128, 64] for smaller variants. Settings were one epoch, batch size 8, learning rate 3×10⁻⁶, warmup ratio 0.1, and NO_DUPLICATES batch sampling, with mixed precision (BF16) on NVIDIA A100 GPUs and a TripletEvaluator on a 1% validation split. The STS phase used 4 epochs, batch size 8, learning rate 2×10⁻⁵, and cosine scheduling with 10% warmup, evaluating every 200 steps with EmbeddingSimilarityEvaluator on Spearman and Pearson correlations.

Stage 2 — Turning encoders into retrievers. The semantically adapted checkpoints became starting points for ColBERT-style training with PyLate on MS MARCO-TR. PyLate's ColBERT module preserves per-token embeddings and applies MaxSim scoring, trained with a contrastive triplet loss (margin = 0.2) on query–positive–negative triples. A ColBERTCollator handles variable-length sequences in batched multi-vector processing, and Weights & Biases was used for monitoring and checkpoint management. The resulting late-interaction models span 4M–150M parameters.

Stage 3 — Making retrieval fast. MUVERA (Multi Vector Retrieval as Sparse Alignment) converts variable-length token embeddings into fixed-dimensional vectors. Given embeddings of dimension d

Authors’ abstract

Neural information retrieval systems excel in high-resource languages but remain underexplored for morphologically rich, lower-resource languages such as Turkish. Dense bi-encoders currently dominate Turkish IR, yet late-interaction models -- which retain token-level representations for fine-grained matching -- have not been systematically evaluated. We introduce TurkColBERT, the first comprehensive benchmark comparing dense encoders and late-interaction models for Turkish retrieval. Our two-stage adaptation pipeline fine-tunes English and multilingual encoders on Turkish NLI/STS tasks, then converts them into ColBERT-style retrievers using PyLate trained on MS MARCO-TR. We evaluate 10 models across five Turkish BEIR datasets covering scientific, financial, and argumentative domains. Results show strong parameter efficiency: the 1.0M-parameter colbert-hash-nano-tr is 600$\times$ smaller than the 600M turkish-e5-large dense encoder while preserving over 71\% of its average mAP. Late-interaction models that are 3--5$\times$ smaller than dense encoders significantly outperform them; ColmmBERT-base-TR yields up to +13.8\% mAP on domain-specific tasks. For production-readiness, we compare indexing algorithms: MUVERA+Rerank is 3.33$\times$ faster than PLAID and offers +1.7\% relative mAP gain. This enables low-latency retrieval, with ColmmBERT-base-TR achieving 0.54 ms query times under MUVERA. We release all checkpoints, configs, and evaluation scripts. Limitations include reliance on moderately sized datasets ($\leq$50K documents) and translated benchmarks, which may not fully reflect real-world Turkish retrieval conditions; larger-scale MUVERA evaluations remain necessary.