A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

AI summary

Overview

Research area: Natural Language Processing / Information Retrieval, specifically Greek-language book search, embedding model comparison, and LLM-assisted retrieval.

Technical level: Intermediate — the paper assumes familiarity with standard IR metrics (nDCG, MRR, Hits@k, R-Precision), sparse vs. dense retrieval, and rank fusion, but explains its setup clearly enough for a reader who knows basic machine learning.

Scope: The paper introduces the CUP benchmark, 868 Greek book catalog records and 104 expert-annotated queries, and uses it to compare sparse, dense, hybrid, and LLM-based retrieval strategies in a real publisher setting (Crete University Press).

What This Paper Is About

Greek is a morphologically rich, relatively low-resource language, and no widely adopted information-retrieval benchmark exists for it — multilingual suites like mMARCO, BEIR, and MIRACL do not support Greek, and Greek-capable suites such as OYXOY and MMTEB do not focus on retrieval. Book search makes this harder because queries range from single keywords to full natural-language requests, and catalog metadata mixes rigid fields (title, author, ISBN) with free-form text (descriptions and tables of contents). The paper builds a graded-relevance benchmark from the Crete University Press catalog and uses it to test which retrieval strategies actually work for Greek books, including whether LLMs can help summarize or filter results.

Key Contributions

1. A new benchmark (CUP): 868 catalog records and 104 expert-annotated queries with graded relevance judgments, covering single-word (30), multi-word (10), noisy (13), category-based (11), reasoning (11), natural language (10), other-language (9), and named-entity (10) queries. Judgments were produced manually by 4 domain experts who examined the entire corpus.
2. A systematic evaluation of 13 embedding models (Greek-specific and multilingual) using Sentence-transformers, including 9 field-aware prefix variations, plus the sparse BM25 baseline.
3. An analysis of hybrid retrieval, combining lexical and semantic signals through greedy-search weighting and Reciprocal Rank Fusion (RRF, smoothing constant k = 60), extended to a two-dense-model configuration.
4. An evaluation of LLM components, specifically LLM-based summarization of sparse tables of contents (five LLMs) and LLM-based post-filtering/reranking of the best hybrid retriever's top-9 results.

Main Findings

• Multilingual models beat Greek-specific ones. The best Greek-specific model (xlm-r) reaches only 0.372 nDCG@9, while several multilingual models exceed 0.50. The authors attribute this to scale and cross-lingual training, plus the scarcity of Greek-specific models.
• Dense retrieval outperforms BM25 overall. The strongest standalone dense model is nemotron* (with field-aware prefixes), at 0.582 nDCG@9, 0.619 nDCG@20, and 0.748 MRR, versus 0.544, 0.567, and 0.652 for BM25.
• Field-aware prefixes help unevenly. They clearly help nemotron (nDCG@9: 0.534 to 0.582; MRR: 0.684 to 0.748) and give smaller gains for bge-m3, but the effect is inconsistent across other models, indicating model dependence.
• Weighted hybrid retrieval is best overall. The configuration s0.4 + (d1_0.6 + d2_0.4)_0.6 (d1 = nemotron*, d2 = nomic*) reaches 0.673 nDCG@9 and 0.810 MRR, improving on the best dense model. Gains are statistically significant (p < 0.01) over most metrics, except MRR versus nemotron*.
• Score-level fusion beats rank-based fusion. RRF variants perform worse than greedy-weighted score fusion.
• Query type determines the winner. BM25 is extremely strong for exact named-entity queries, particularly authors, where it reaches perfect nDCG@9. Hybrid retrieval reaches perfect performance on named-title queries (versus 0.765 for BM25) and outperforms both BM25 and dense models on concept queries. Dense-only retrieval can be preferable for cross-lingual and reasoning queries, where nemotron* leads nDCG@9.
• Hybrid helps the hardest realistic queries. For natural-language queries, nDCG@9 improves from 0.374 (BM25) to 0.553 (best hybrid); for noisy queries, from 0.488 to 0.672, showing tolerance to misspellings and missing diacritics.
• Six queries defeat every configuration. Six noisy, natural-language, multi-word concept, and category queries return no relevant books in the top-9 across all configurations.
• LLM TOC summaries beat raw TOC indexing. Summaries of roughly 500 words improve retrieval over indexing the raw toc field, with lower length variance but reduced lexical diversity. deepseek is more verbose; krikri and euroLLM are more concise. euroLLM achieves the best nDCG@9 and Hits@{3,9}, and together with krikri produces the most linguistically correct Greek summaries. meltemi scores well only because it fails to follow the prompt and leaks contextual metadata (author, title, description) into the summary.
• LLM post-filtering improves early precision at high cost. Given the top-9 results from the best hybrid retriever with full metadata, Qwen3-8B performs best overall, improving Hits@1, nDCG@3, and MRR (0.821), but returns shorter filtered lists, lowering Hits@9 relative to retrieval-only systems. deepseek, euroLLM, and krikri are worse than the best hybrid, and meltemi fails due to poor prompt adherence.
• Prompt adherence is measurable. Invalid or missing ID rates were 75% for meltemi, 3.85% for euroLLM/krikri, 2.88% for deepseek, and 1.92% for qwen3.
• The LLM gains are expensive. Qwen3's post-filtering gains come at an average of 92.5 seconds of inference per query in the authors' setting.
• Every field contributes. An ablation shows that removing any field degrades performance, with the largest drop when the author field is removed (0.606 versus 0.675 nDCG@9) and the smallest for categories.
• No single strategy dominates. BM25 is strongest for exact-match queries, dense models excel in reasoning and cross-lingual scenarios, and hybrid retrieval provides the best overall performance.

Methodology in Plain English

The researchers took a real publisher's catalog — 868 books across sciences (STEM), history and social sciences, humanities, and arts — and built a search benchmark around it. Most content is Greek, with a small number of English-only, bilingual (Greek-English, Greek-French), and trilingual records. Documents were represented over five fields of equal weight: title, author, categories+tags, content (the publisher's description, averaging about 225 words, maximum 1,154), and toc (raw table of contents, averaging 226 words but ranging from empty to nearly 2,000 words; 342 books, 39% of the corpus, have TOCs exceeding 200 words). Preprocessing normalized titles and author names, aggregated categorical metadata, and stripped boilerplate from descriptions.

For lexical search they used a BM25 retriever via the rank_bm25 library inside LangChain, with lowercasing, accent stripping, stemming, and stopword removal. For semantic search they encoded documents with 13 Sentence-transformer embedding models, following each model's recommended prompting or formatting convention, plus 9 variations using field-aware prefixes (for example, prefixing the author field with "Ο συγγραφέας του βιβλίου είναι …"). Embeddings were indexed with FAISS.

They then combined lexical and semantic signals: hybrid weights were tuned by greedy search, and they also tested Reciprocal Rank Fusion over multiple models. Five LLMs (Meltemi-7B, Llama-Krikri-8B, EuroLLM-9B, Qwen3-8B, and DeepSeek-R1-Qwen3-8B) were used in two distinct roles — summarizing sparse tables of contents to enrich document representations, and post-filtering/reranking the top results of the best hybrid retriever. Evaluation used nDCG@{9,20}, MRR, Hits@{1,3,9}, and R-Precision, computed with the rank-eval library. Results at k = 20 are not reported for LLM filtering and reranking because of GPU context limits (AMD Radeon AI PRO R9700 32GB) and model restrictions.

Why This Matters

The work fills a concrete gap: the authors state that no widely adopted information-retrieval benchmark for Greek existed, and that Greek is absent from mMARCO, BEIR, and MIRACL, while OYXOY and MMTEB do not focus on retrieval. CUP therefore gives researchers a real-world, graded-relevance testbed spanning lexical, semantic, noisy, and cross-lingual queries, and it establishes a useful empirical pattern — multilingual scale currently outweighs language-specific adaptation for Greek retrieval.

Real-world applications:

• Publisher and bookstore catalogs: the findings give concrete guidance on when keyword search suffices (named-entity and author lookups) and when semantic or hybrid search is needed (natural-language, noisy, concept, and cross-lingual queries).
• Multilingual e-commerce and digital libraries: the same sparse-dense-hybrid trade-offs apply anywhere users search in a morphologically rich language with inconsistent metadata.
• Cross-lingual search: the benchmark includes English queries against a mostly Greek catalog, which mirrors real behavior of users searching in a second language.
• Metadata enrichment pipelines: LLM summarization of sparse tables of contents is shown to be a practical way to improve retrieval over raw, inconsistently formatted structured text.

Industry relevance: the paper quantifies a real deployment constraint. LLM post-filtering improves early precision but costs an average of 92.5 seconds per query, which the authors explicitly flag as too costly for a real-time setting. That trade-off, plus the prompt-adherence failure modes (a 75% invalid or missing ID rate for one model), is directly relevant to teams deciding whether to put an LLM in a production search loop.

Future Directions

• Extending the benchmark with more records and queries, as the authors state in their conclusion.
• Further evaluating LLM-based reranking in addition to the post-filtering tested here.
• Evaluating answer generation, and doing so in interactive settings rather than single-shot retrieval.
• Open questions the results raise: why field-aware prefixes help some embedding models but not others, whether LLM summarization quality can be raised without leaking metadata the way meltemi does, and how to make LLM post-filtering affordable enough for real-time use. The six query types that no configuration solved also remain unaddressed.

Target Audience

Researchers and practitioners in information retrieval and multilingual NLP, especially those working on low-resource or morphologically rich languages. It is also directly useful to engineers building search for publishers, libraries, or catalogs with mixed structured and free-form metadata, and to teams weighing whether LLM-based summarization or reranking is worth its inference cost. Readers without background in IR evaluation metrics will need some preparation, since the paper reports nDCG, MRR, Hits@k, and R-Precision without introductory explanation.

Authors’ abstract

We present CUP, a Greek book retrieval benchmark consisting of 868 catalog records and 104 expert-annotated queries with graded relevance judgments. We evaluate sparse (BM25), dense (sentence-transformers), hybrid, and LLM-assisted retrieval methods in this book-search setting. Multilingual embeddings outperform Greek-specific models, while hybrid retrieval performs best overall. A query-level analysis shows that BM25 excels at named-entity queries, while dense and hybrid methods improve natural-language, noisy, cross-lingual, and concept queries. Field-aware prompting has model-specific effects, while LLM TOC summarization improves TOC-only retrieval and LLM post-filtering improves early-stage retrieval at a high cost. Overall, CUP enables real-world evaluation of Greek retrieval across lexical, semantic, noisy, and cross-lingual queries.