SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

AI summary

Overview

Research area: Natural language processing — text embedding evaluation and efficient multilingual model adaptation for a low-resource language (Slovak).

Technical level: Intermediate. The benchmark design is easy to follow, but interpreting per-task-type score tables, V-measure, nDCG, and equivalence testing assumes some familiarity with embedding evaluation.

Scope: This paper introduces SkMTEB, a 31-dataset Slovak text embedding benchmark covering 7 task types, evaluates 31 embedding models on it, and adapts Multilingual E5 into compact Slovak-specific models via vocabulary trimming and fine-tuning.

What This Paper Is About

Text embedding models are now basic infrastructure for search, retrieval-augmented generation (RAG), clustering, and classification, but evaluation evidence and efficient models are concentrated in high-resource languages. Slovak — a West Slavic language with approximately 5 million speakers — had almost no embedding evaluation infrastructure: the existing MMTEB benchmark covers it with only 8 tasks, and the earlier skLEP benchmark targets natural language understanding rather than embedding tasks. The paper's goal is twofold: build a deep Slovak embedding benchmark, and show that compact, locally deployable Slovak embedding models can be trained with modest resources.

Key Contributions

1. SkMTEB, the first comprehensive Slovak embedding benchmark — 31 datasets across 7 task types, nearly 4× the depth of existing multilingual benchmark coverage for Slovak, with only 6 datasets overlapping MMTEB.
2. Dataset construction and adaptation — to overcome the severe shortage of Slovak datasets, the authors adapted multiple existing datasets to new tasks and introduced seven brand-new datasets. Coverage spans domains (medical, fact-checking, parliamentary), task formulations (summarization-as-retrieval, URL-based clustering), and temporal ranges from 2000 to 2025.
3. A 31-model baseline evaluation — spanning compact, mid-sized, and large open-weight models plus proprietary APIs, grouped into small (<130M), base (130M–350M), large (≥350M), and API-access categories.
4. Vocabulary-trimmed Slovak E5 models — e5-sk-small (45M parameters) and e5-sk-large (365M), built by applying vocabulary trimming and targeted fine-tuning to Multilingual E5, with ablations isolating trimming, fine-tuning, and prompt usage. All models, datasets, and code are released openly.

Main Findings

• Instruction-tuned large multilingual models lead. multilingual-e5-large-instruct achieves the highest overall score (77.49), followed closely by gemini-embedding-001 (77.23). These models excel particularly in classification and clustering, where they outperform smaller alternatives by significant margins.
• Slovak-specific NLU models transfer poorly to embedding tasks. slovakbert-skquad-mnlr (125M) and slovakbert-sts-stsb (125M), trained for NLU, underperform multilingual alternatives. The authors' own sturovec-base (125M) scored 68.99, falling short of the unfine-tuned multilingual-e5-small baseline (70.32) despite using 1.4M training examples.
• Scale alone shows diminishing returns for Slovak. jina-embeddings-v4 (3.8B, 72.44) trails snowflake-arctic-embed-l-v2.0 (568M, 72.54) and nomic-embed-text-v2-moe (330M, 72.58), and only narrowly edges multilingual-e5-base (278M, 72.39).
• Vocabulary trimming preserves performance while shrinking models. Trimming reduced multilingual-e5-small from 118M to 45M parameters (62% reduction) with a +0.13 average gain, and multilingual-e5-large from 560M to 365M (35% reduction) with +0.31.
• Compact trimmed models match proprietary APIs. e5-sk-small (70.56) performs on par with text-embedding-3-small (70.48), and e5-sk-large (74.70) is comparable to text-embedding-3-large (75.07). TOST equivalence testing confirms practical equivalence, with 90% confidence intervals for both comparisons falling within ±2 points.
• Prompts help small models more than large ones. Adding query: / passage: prefixes improved E5-small by +0.51 (70.56 → 71.07) but E5-large by only +0.02 (74.70 → 74.72), suggesting explicit query-passage distinction particularly benefits models with limited capacity.
• Cross-lingual transfer survives aggressive trimming. On six bitext mining tasks, the maximum absolute change was 0.92 F1 for the small model and 0.14 F1 for the large model, with average changes of 0.25 F1 and 0.04 F1 respectively. Several pairs marginally improved after trimming.
• Task difficulty varies widely. Bitext mining is largely solved, with most models reaching F1 above 90. Clustering remains challenging, with V-measure ranging from 17 to 50. STS favors models with explicit similarity objectives — jina-embeddings-v3 reached 89.82.
• The 45M model reaches 91% of the best 560M model's performance.

Methodology in Plain English

The authors first assembled a benchmark by collecting every Slovak dataset they could find and then filling gaps. Where existing Slovak data existed but only for other tasks, they reformulated it — for example, turning news summarization datasets into retrieval tasks by using article abstracts as queries to find full documents in collections of 200k+ and 80k articles. They also created seven new datasets, including pharmacy question-answering reranking sets scraped from two Slovak pharmacy forums and processed to remove personal names and duplicates.

For evaluation, they ran 31 existing embedding models with their default configurations and recommended preprocessing, measuring each with the standard metric for its task type — nDCG@10 for retrieval, MAP for reranking, accuracy for classification, V-measure for clustering, F1 for bitext mining, average precision for pair classification, and Spearman correlation for STS. Results are reported both as an average across all tasks and as an unweighted average across task types.

For model adaptation, they fine-tuned on higher-quality skLEP data: SK-SQuAD (72K query-context pairs), NLI translated from XNLI (393K pairs), STS from GLUE STS-B (6K pairs), and RTE from GLUE (2.5K pairs). They initially included 967K Slovak Web QA pairs from WebFAQ and MFAQ, but dropped them because randomly sampling answers from the same domain as hard negatives did not consistently provide meaningful contrastive signal. Before fine-tuning they applied vocabulary trimming, keeping 60K of 250K tokens selected by frequency in the FineWeb2-Slovak corpus. Training used mean pooling, a max sequence length of 256, batch size 32, learning rate 2×10⁻⁵, 3 epochs, a single NVIDIA H100 GPU, and random seed 42 — finishing in under one hour per model variant.

Why This Matters

This is a template for how a low-resource language can get serious embedding evaluation and practical models without a large compute budget. The authors explicitly frame four transferable lessons: NLU-tuned monolingual encoders underperform on embedding tasks, so language-specific embedding evaluation is needed even where NLU benchmarks exist; vocabulary trimming generalizes from NLU to embedding models with 35–62% parameter reductions and negligible degradation; 4B–8B parameter embedding models rarely beat 500M–600M counterparts on a single target language; and the whole pipeline can be replicated in under one GPU-hour per adapted model.

Real-world applications:

• Semantic search over Slovak text, including public-sector document search.
• Retrieval-augmented generation (RAG) pipelines, where locally deployable embeddings avoid per-call API costs.
• Clustering and classification of Slovak news, reviews, and social media content.
• Accessibility tooling and content moderation over Slovak text.

Industry relevance: The headline result for practitioners is that a 45M-parameter open-weight model can match text-embedding-3-small on Slovak while running locally, enabling higher throughput and removing API dependency. Because the benchmark skews toward news, Wikipedia, and web content, companies with legal, medical, or technical Slovak corpora should validate performance on their own domain before deploying. The paper also notes that retrieval and clustering over Slovak political discourse could enable surveillance or political profiling, recommending context-appropriate governance.

Future Directions

• Natively authored Slovak data. A significant portion of the benchmark derives from translated datasets (NLI, STS, RTE), which may carry translationese artifacts rather than authentic Slovak use.
• Broader domain coverage. Legal, medical, and technical Slovak are underrepresented, as the current benchmark skews toward news, Wikipedia, and web content.
• Wider model and language-pair evaluation. Computational constraints limited evaluation of the largest models (8B+ parameters) across all tasks, and trimming was only tested on Slovak-English and Slovak-Czech pairs — not on non-Slavic, non-English pairs or code-mixed, multi-language sentences.
• Keeping evaluations current. Both the benchmark and model evaluations are a temporal snapshot in a rapidly evolving field; the authors invite ongoing community contributions.

Target Audience

Researchers and engineers working on embeddings or retrieval for under-resourced languages, especially those interested in benchmark construction and compute-efficient model adaptation. Also useful for practitioners who need deployable Slovak semantic search or RAG components and want to know which existing models actually perform well, and for teams in other low-resource languages looking for a replicable pipeline — benchmark construction, broad evaluation, vocabulary trimming, and fine-tuning — that runs in under a GPU-hour per adapted model.

Authors’ abstract

We introduce SkMTEB, the first comprehensive MTEB-style text embedding benchmark for Slovak, a low-resource West Slavic language, comprising 31 datasets across 7 task types -- nearly 4$\times$ the depth of existing multilingual benchmark coverage for Slovak. Our evaluation of 31 embedding models reveals that large instruction-tuned multilingual models achieve the strongest performance, while existing Slovak-specific models trained for NLU tasks transfer poorly to embedding tasks. To address the need for efficient, locally-deployable Slovak embeddings, we develop \texttt{e5-sk-small} (45M parameters) and \texttt{e5-sk-large} (365M) by applying vocabulary trimming and fine-tuning to Multilingual E5 models. Despite size reductions of up to 62\%, our open-source models achieve competitive performance with proprietary APIs while remaining locally deployable for semantic search and retrieval-augmented generation (RAG). We release the benchmark, models, datasets, and code openly, hoping our approach offers a replicable path for other under-resourced languages.