HUME: Measuring the Human-Model Performance Gap in Text Embedding Tasks

AI summary

Overview

Research area: Natural Language Processing — evaluation methodology for text embedding models (benchmarking, human annotation, LLM-as-annotator).

Technical level: Intermediate. No model architecture or training is involved; the paper is about measurement, benchmark design, and interpreting scores. Readers need only a working sense of what an embedding model does and what benchmarks like MTEB are.

Scope: The paper introduces HUME (Human Evaluation Framework for Text Embeddings), which measures how well human annotators perform on 16 MTEB embedding datasets and compares them directly to 13 embedding models and nine LLMs acting as annotators.

What This Paper Is About

Embedding benchmarks such as MTEB report model scores against theoretical maxima (for example MAP = 1.0), which assume perfect human consensus, so it is unclear whether a given score is strong, mediocre, or beyond what people actually achieve. Because measuring human performance on embedding tasks is difficult, such comparisons are rarely made. HUME fills that gap by collecting human annotations on a downsampled, matched set of MTEB tasks so that humans and models are scored on identical instances with identical metrics.

Key Contributions

1. A generalizable human evaluation framework for embedding tasks. HUME adds task-specific annotation interfaces, principled dataset sampling, a standardized results format, and metrics aligned with MTEB across four task categories (reranking, classification, clustering, STS) and 16 datasets.

2. Empirical human performance baselines. The authors measure human performance across 16 MTEB datasets covering high-resource languages (English, Arabic, Russian) and lower-resource languages (Norwegian Bokmål, Danish), reporting per-task results, inter-annotator agreement, and 95% confidence intervals.

3. A direct human-versus-model comparison. Human performance is placed alongside 13 embedding models spanning 22M–7B parameters, encoder- and decoder-based architectures, instruction-tuned and standard models, and English-only versus multilingual capability — with humans ranking 4th overall at 77.6 against the best model at 80.1.

4. An LLM-as-annotator benchmark plus dataset quality diagnostics. Nine LLMs (GPT-5 full and mini, GPT-4.1 full and mini, Gemini 2.5 Flash, Mistral Small-24B-Instruct, and Qwen3 30B, 32B, and Coder-30B) are evaluated on the same instances, and the human agreement data is used to flag specific MTEB datasets whose reliability is compromised.

Main Findings

• Humans are competitive but not dominant. Across 16 tasks, human annotators rank 4th of 13 embedding models with an overall score of 77.6, versus 80.1 for the best model (jasper_en_vision_language_v1). Humans never form a uniform ceiling or floor, occupying a middle ground that shifts by task category, language, and dataset quality.

• Humans usually beat the median model but rarely the best one. Across 26 task-language pairs, humans exceed median model performance in 61.5% of tasks and match the best models in only 15.4%. Models perform outside human confidence intervals in 14 of 26 tasks (p < 0.05), often on datasets with low inter-annotator agreement.

• Performance varies sharply by task category. Classification averages 70.3 for humans (range 45.8 on emotion classification with κ = 0.39, to 95.0 on Arabic sentiment), with models generally higher (best 87.1). Clustering averages 67.4 V-measure with extreme spread — 97.6 (ARI = 0.91) on WikiCities versus 49.2 (ARI = −0.001) on ArXiv — with models consistently higher (best 85.1%). Reranking is strong for humans at 87.2 average MAP with high agreement (ρ = 0.64–0.85), versus a best model of 96.4. STS averages 83.2 Spearman, from 91.2 on STS12 down to 58.7 on STS22-Russian, versus a best model of 88.5.

• Humans excel on non-English tasks. Against the best models humans win only 15% of tasks overall, but 62% against median models. Against best models the win rate is 29% on multilingual tasks versus 0% on English-only tasks. Arabic shows the largest advantage: 67% win rate against the best models and 100% against the mean, with the biggest gap in semantic similarity (67.5% human vs 40.9% best model, a 26.6-point margin). Russian and Norwegian sentiment also favor humans at 92.5% and 85.0%.

• "Superhuman" model performance often tracks low human agreement. Emotion classification reaches only fair agreement (κ = 0.39, 52.1% consensus) yet the best model scores 87.1% against 45.8% for humans; ArXiv clustering has essentially zero human agreement (ARI = −0.001) while a model reaches 84.6% against 49.2% for humans. By contrast, high-agreement tasks such as reranking (ρ = 0.64–0.85) and toxicity classification (κ = 0.55, 77.8% consensus) look like genuine evaluation targets.

• LLMs do not yet replace human annotators. Across 19 task-language pairs (clustering excluded), the best LLM (GPT-4.1-mini) reaches 76.1% versus 81.2% for humans. Classification is near parity (GPT-5: 78.9% vs humans 79.1%), but reranking shows a 10-point deficit (humans 88.3% vs best LLM Mistral-Small 78.0%), precisely where human agreement is highest. On STS, humans reach 76.5% versus 75.0% for the best LLM. Embedding models lead every category (Classification 80.3%, Clustering 79.1%, Reranking 94.8%, STS 77.1%).

• Human and LLM difficulty patterns only partly overlap. The Spearman rank correlation between human and LLM performance across the 19 task-language pairs is moderate and positive (ρ = 0.52, p < 0.05, n = 19), indicating shared but not identical challenges.

• Specific datasets are recommended for replacement. The paper names emotion classification, ArXiv clustering, and STS22-Russian (systematic parsing artifacts) as unreliable evaluation targets, and proposes that leaderboards report human agreement alongside model scores — a model at 85% on emotion classification (185% of human performance, κ = 0.39) is a fundamentally different result from 85% on reranking (97% of human performance, ρ = 0.75).

Methodology in Plain English

The team built an annotation pipeline inside Argilla that mirrors what MTEB does, but with people in the model's place. They picked 16 MTEB datasets to cover multiple languages, domains (news, social media, encyclopedias, scientific literature, forums), construction methods (curated versus synthetic), and difficulty levels, then downsampled each so annotation was feasible: reranking 20–49 queries, classification 40–48 examples, clustering 30 items, and STS 30–50 pairs.

Instructions were written to match the original task definitions exactly — the same label sets for classification, the same 1–5 scale for STS — so human and model scores would be directly comparable. Instructions were piloted on a small subset first to resolve ambiguities. Direct human evaluation of retrieval is impractical, so the authors use reranking as a human-evaluable proxy, giving annotators only the top-k candidates.

Annotators were NLP practitioners with native or near-native proficiency in the evaluated languages, recruited for cultural and language diversity, and they worked independently with no access to ground truth or model predictions. English tasks got two annotators so agreement could be computed; multilingual tasks got a single language-expert annotator. Agreement was measured with Fleiss' kappa (classification), pairwise ARI (clustering), pairwise Spearman (STS), and mean Spearman/Kendall's tau (reranking). The 13 embedding models were then evaluated on exactly the same downsampled instances with the same metrics (MAP, accuracy, V-Measure, Spearman), and significance was assessed with 95% confidence intervals via Wilson Score Intervals for accuracy and Fisher z-transformations for correlations. Nine LLMs were prompted with the identical human instructions and the same evaluation metrics; clustering was excluded from LLM testing because structured cluster assignments were too difficult to elicit from generative models.

Why This Matters

The paper reframes benchmark scores as interpretable relative to human consensus rather than to a theoretical maximum, and treats human agreement as a validity threshold for a task. When models badly exceed human agreement on a low-agreement task, the suggested reading is that the benchmark has lost descriptive power — it may be rewarding reproduction of annotation artifacts, which the authors call "blind optimization." This matters for how embedding progress is measured, which datasets survive in future MTEB iterations, and how much confidence to place in LLM-generated annotations.

Real-world applications:

• Search and retrieval systems: reranking is the proxy for information retrieval here, and the persistent gap (96.4% best model vs 87.2% humans, with strong human agreement) points to a real modeling problem rather than label noise.
• Content moderation: toxicity classification shows moderate agreement (κ = 0.55) with 77.8% annotator consensus, making it one of the more dependable targets for building filters on.
• Multilingual and culturally specific products: the largest human advantages appear in Arabic semantic similarity and non-English sentiment (95.0% Arabic, 92.5% Russian, 85.0% Norwegian), which are exactly the settings where a deployed model would misjudge user intent.
• Benchmark and leaderboard design: teams building internal evaluations can use the framework to check whether their task has clear ground truth before spending engineering effort optimizing against it.

Industry relevance: organizations that pick embedding models from public leaderboards can use these baselines to judge whether a headline number reflects genuine capability or a noisy label set. The paper's recommendation — report human agreement next to model scores — is a directly actionable change for anyone publishing or consuming an embedding leaderboard, and the finding that LLM annotation introduces systematic bias on fine-grained relevance judgments cautions against replacing human evaluation to cut costs.

Future Directions

• Scale up sample sizes. The study prioritized breadth over depth, with 20–50 instances per task. Larger samples would characterize human judgment distributions more robustly than the current significance analyses allow.

• Use trained expert annotators. Annotators were average or above-average raters without task-specific training, and original dataset guidelines were sparse. Experts would likely score higher on technical tasks, and clearer alignment with original annotation procedures remains unresolved.

• Understand why the gaps exist. The paper quantifies where models diverge from humans but does not explain the causes — training data coverage, domain or cultural bias, linguistic variability in low-resource settings. Model training corpora are often undocumented, which limits this analysis.

• Study task design principles directly. The authors did not systematically test how specification clarity versus meaningful challenge affects human agreement and model discrimination, and they call for research that avoids both ambiguity (which suppresses human agreement) and oversimplification (which destroys discriminative power). They also suggest future benchmarks could preserve and leverage annotator disagreement rather than collapsing it to a single gold label.

• Develop hybrid annotation pipelines. Given that LLMs trail humans most on reranking, the suggested direction is using LLMs for initial large-scale annotation while reserving human judgment for high-agreement tasks and uncertain cases.

Target Audience

Embedding model developers and benchmark maintainers will get the most from this paper, since it supplies reference numbers and a concrete deprecation list for MTEB datasets. NLP evaluation researchers interested in human baselines, inter-annotator agreement, and LLM-as-annotator validity will find the framework and the negative result on LLM proxies directly useful. Applied teams choosing an embedding model for search, moderation, or multilingual products benefit from the guidance on interpreting leaderboard scores and on where current models still lag human judgment. Readers who want the full per-task tables, agreement analyses, model lists, and LLM breakdowns should consult the paper's appendices (A, B, C, D, E, F, G, H, I), which are referenced throughout but truncated in the content summarized here.

Authors’ abstract

Comparing human and model performance offers a valuable perspective for understanding the strengths and limitations of embedding models, highlighting where they succeed and where they fail to capture meaning and nuance. However, such comparisons are rarely made, as human performance on embedding tasks is difficult to measure. To fill this gap, we introduce HUME: Human Evaluation Framework for Text Embeddings. While frameworks like MTEB provide broad model evaluation, they lack reliable estimates of human performance, limiting the interpretability of model scores. We measure human performance across 16 MTEB datasets spanning reranking, classification, clustering, and semantic textual similarity across linguistically diverse high- and low-resource languages. Humans achieve an average performance of 77.6% compared to 80.1% for the best embedding model, though with substantial variation: models reach high performance on some datasets while struggling on notably low-resource languages. Our human annotations also reveal multiple dataset issues. We additionally benchmark nine LLMs as annotators on reranking, classification, and STS tasks, finding that they fall short of human performance (76.1% vs. 81.2%) despite offering scalability advantages. We provide human performance baselines, insights into task difficulty patterns, and an extensible evaluation framework that enables a more meaningful interpretation of results and informs the development of both models and benchmarks. Our code, dataset, and leaderboard are publicly available at https://github.com/embeddings-benchmark/mteb.