Peeling the Matryoshka: Extracting Ontological Structure from Retrieval Embeddings

AI summary

Overview

Research area: Natural Language Processing — taxonomy/ontology construction, representation learning, and structural probing of pre-trained embeddings.

Technical level: Intermediate. The paper is readable without deep mathematical background, but familiarity with sentence embeddings and vector similarity helps.

Scope: The paper reports that Matryoshka Retrieval Learning (MRL) embedding prefixes encode coarse-to-fine taxonomic granularity, and introduces PEEL, a 98K-parameter probe that decodes parent–child (is-a) structure from frozen retrieval embeddings using only a concept vocabulary.

What This Paper Is About

Building domain taxonomies — hierarchies of concepts linked by is-a relations — is expensive: clustering methods need per-corpus tuning and produce unstable hierarchies, while LLM-based pipelines need costly task-specific training that does not transfer across domains. The authors ask whether embeddings trained purely for retrieval already contain hierarchical signal that can be exploited, and find that in Matryoshka Representation Learning (MRL) embeddings, where every leading prefix of the vector is itself a usable coarser representation, prefix length aligns with the coarse-to-fine granularity of gold taxonomies. They then build PEEL to test whether that regularity is operationally usable for recovering taxonomic DAGs.

Key Contributions

1. Discovery. An empirical finding that pre-trained MRL embeddings encode taxonomic structure in their learned early prefix dimensions, isolated by shuffled-dimension and reverse-dimension controls and replicated across three architecturally distinct open-source MRL models (Nomic, jina-v3, Qwen3).

2. Demonstration. PEEL (Peeling Embeddings to Extract Latent ontologies), a minimal probe that decodes taxonomy DAGs via minimum spanning arborescence without any task-specific embedding training — a 98K-parameter projection over frozen embeddings.

3. Evaluation. On two OLLM benchmarks, PEEL recovers taxonomic DAGs from a concept vocabulary alone with high structural fidelity on the primary structure-sensitive metrics (Graph F1 .885 on wiki-ol and .931 on arXiv-ol), with the document-fed OLLM pipeline reported as a cross-task reference rather than a same-conditions comparison.

4. Multi-resolution decoding. Trees extracted at different prefix lengths are non-nested (edge-set Jaccard below .18 between all dimension pairs, with 99% of nodes changing parents), showing that a single frozen embedding yields distinct hierarchies at different granularities.

Main Findings

• Prefix length tracks taxonomic depth. kNN label purity (k=10) from MRL prefix truncation outperforms both a shuffled-dimension control (ordering destroyed, content preserved) and a last-k reverse control across Nomic-embed-text-v1.5 (137M, BERT), jina-embeddings-v3 (572M, XLM-R), and Qwen3-Embedding (600M, decoder), with the advantage concentrated in the early truncated dimensions. At full dimensionality the prefix equals the complete embedding, so all three coincide by construction.

• Effect sizes grow with depth. A supporting one-way ANOVA over every embedding dimension reports mean η² that is small-to-medium at Depth 1 (.04–.07), medium at Depth 2 (.13–.18), and Cohen's-large at Depth 3 (.34–.40) on Wikipedia; the shallower arXiv taxonomy shows the same direction (Depth 1 .04–.08, Depth 2 .12–.20). The depth ordering holds without exception across all six model-by-dataset sequences.

• Directional consistency is statistically strong. Per-condition win counts for MRL prefix truncation over the shuffled control give combined one-sided sign tests of 42/50 (p = 5.8 × 10⁻⁷) for Nomic, 50/55 (p = 1.1 × 10⁻¹⁰) for jina-v3, and 49/55 (p = 9.1 × 10⁻¹⁰) for Qwen3; every per-model p is below 10⁻⁶.

• High structural fidelity from vocabulary alone. PEEL reaches Graph F1 .885 and Cont. F1 .551 on wiki-ol (8,033 concepts, 14,673 gold edges at depth 3, supervised with an edge-level 80/10/10 split), and Graph F1 .931 and Cont. F1 .604 on arXiv-ol (61 categories, depth 2) zero-shot from the wiki-ol-trained probe.

• Comparison against OLLM. The document-fed OLLM pipeline scores Graph F1 .644 and Cont. F1 .500 on wiki-ol, and .633 / .357 on arXiv-ol (the latter after fine-tuning Mistral-7B on 2,048 additional arXiv samples). The authors emphasize the comparison is asymmetric: PEEL receives strictly less input, and OLLM performs the broader task of joint concept discovery and structure induction.

• Label-string metrics favor LLM methods, as expected. Memorisation tops Lit. F1 (.134) by replaying training edges verbatim, and OLLM leads Fuzzy F1 (.915) on wiki-ol; PEEL scores Lit. F1 .111 and Fuzzy F1 .877 there, and Lit. F1 .298 / Fuzzy F1 .748 on arXiv-ol. PEEL's Motif Dist. is .000 on wiki-ol and .011 on arXiv-ol versus OLLM's .080 and .097.

• Truncation matters and matches the prefix gradient. Cont. F1 peaks at d=512 (.551) and drops at full dimensionality (.479 at d=768), while Graph F1 stays in .879–.906 across dimensions, peaking at d=768. Even a d_out=32 probe (24K parameters) yields Graph F1 .870 and Cont. F1 .517.

• Prefixes unfold fine-grained structure progressively. The Mathematics subtree grows from 1 child at d=64 to 19 at d=256 and 68 at d=768, spanning 12 levels; Geography goes from 1 child to 38 across depth 10, and Science from 3 children at depth 1 to 51 across depth 17.

• Raw embeddings already carry a weak generality signal. Before any probe training, a training-free measure over raw prefixes scores 69.1% on gold parent–child pairs and 91.3% on ancestor–descendant pairs at depth gap ≥ 2, while equal-granularity siblings sit at chance (50.0%).

• Structural similarity exceeds exact-edge match. On arXiv-ol, PEEL reproduces 18 of 60 gold edges exactly (30%) yet achieves Graph F1 .931, reflecting the difference between exact edge matching and SGConv-based structural similarity.

Methodology in Plain English

The authors first check whether the hierarchy signal is really there. They take three open-source MRL encoders and, for each concept, truncate the embedding to its first k dimensions — a shorter prefix should give a coarser representation. They measure whether a concept's nearest neighbours (k=10) share its taxonomic label, and compare against two deliberately broken versions: one where all dimensions are randomly shuffled before truncation (same information, no learned ordering) and one where the last k dimensions are used instead of the first. If the real prefix beats both, the ordering is doing the work. A one-way ANOVA on η² across taxonomy depths serves as a supporting cross-model consistency check.

For PEEL itself, the recipe has three deliberately simple stages. First, a single linear layer (768 → 128) with a ReLU maps frozen embeddings into a non-negative "order-embedding" space, trained with an order-embedding loss that pushes a parent's coordinates to be no larger than its child's in every dimension. The probe is trained only on wiki-ol gold parent–child edges (80/10/10 split, negatives at a 10:1 ratio mixing random nodes, siblings, and reversed pairs, margin 4.0, Adam at 5 × 10⁻⁴, batch 512, 10 epochs, roughly 4 seconds on one GPU). During training, embeddings are randomly truncated to one of {64, 128, 256, 384, 512, 768} dimensions and zero-padded back to 768, so a single probe works at every resolution.

Second, a pairwise "violation score" measures how badly one concept violates the ordering with respect to another. Running the Chu-Liu/Edmonds minimum spanning arborescence algorithm over these scores produces an optimal rooted tree with exactly N−1 edges and no threshold to tune. Third, because real ontologies are polyhierarchical, the tree is expanded into a DAG by greedily adding the lowest-violation non-tree edges while keeping the graph acyclic, up to a budget.

Hyperparameters (d=512, expansion budget 15,000, candidate count k=10) are selected on wiki-ol validation. For arXiv-ol the wiki-trained probe is applied unchanged with d=768, k=20, and tree-only extraction; no arXiv-ol data is used for training, validation, or hyperparameter selection. Evaluation uses the five OLLM metrics: Lit. F1, Fuzzy F1 (all-MiniLM-L6-v2, τ=0.436), Cont. F1, Graph F1, and Motif Distance.

Why This Matters

Impact on research. The paper reframes taxonomy construction as a decoding problem rather than a generation or training problem. It extends the structural-probing paradigm — previously aimed at syntax — to taxonomic structure, and inverts the usual assumption that hierarchy requires hierarchy-aware training. The authors contrast their "hierarchy discovery" with the "hierarchy injection" of prior work that custom-trains Matryoshka embeddings with modified contrastive losses, and note that a 98K-parameter probe avoids the 22M–109M-parameter encoder retraining required by hyperbolic approaches such as HiTs, a three-order-of-magnitude reduction. A retrieval-efficiency objective, they argue, incidentally produces a multi-resolution taxonomic geometry.

Real-world applications:

• Building or refreshing domain taxonomies (biomedicine, law, and other specialized fields named as future directions) from an existing term list without a document corpus.
• Organizing product catalogs and e-commerce category trees, where a curated vocabulary already exists but parent–child structure does not.
• Structuring internal knowledge bases and document-management systems by supplying a concept list from any front-end — an LLM extractor, a corpus miner, or a curated vocabulary — to the structure-induction stage.
• Retrieval and search infrastructure, since the same MRL embeddings used for search can be probed for hierarchy at multiple granularities from a single forward pass, with no retraining or per-resolution recomputation.

Industry relevance. The method runs in seconds on a single GPU and requires no LLM fine-tuning, making it cheap relative to pipelines that fine-tune Mistral-7B-scale models. Because MRL embeddings are widely deployed in production retrieval systems, the hierarchical signal described here is already present in models many organizations are running. The authors also note the environmental footprint advantage of avoiding large-model fine-tuning, while cautioning that constructed taxonomies inherit biases from pre-training corpora and should be audited in sensitive domains.

Future Directions

• Broader domain coverage. Evaluation spans only two OLLM benchmarks (deep polyhierarchy and shallow tree). Extension to specialized domains such as biomedicine and law is named as a natural next step.

• How far zero-shot transfer reaches. Validated transfer covers target ontologies no deeper than the source domain; the authors state that how far this reach extends is genuinely open.

• End-to-end integration. Pairing PEEL with a corpus-based concept-discovery front-end is proposed as the natural route to a full taxonomy-construction pipeline, with the structure-induction stage left untrained.

• Head-to-head comparison with hyperbolic methods. A direct benchmark comparison between HiTs and PEEL on the OLLM taxonomies is explicitly left for future work, along with the broader question of downstream utility (retrieval, question answering), which the paper places outside its scope.

Target Audience

Researchers and practitioners in NLP and knowledge engineering who work on taxonomy or ontology induction, representation learning, or probing of pre-trained models. It is also useful for engineers building search and retrieval systems on MRL embeddings who want cheap hierarchical structure without a large-model pipeline, and for readers interested in what retrieval objectives implicitly encode. The paper is most rewarding for those comfortable with embedding similarity and evaluation metrics, but its central claim — that prefix length aligns with taxonomic granularity — is stated plainly enough for newcomers to follow.

Authors’ abstract

Taxonomy construction is essential for knowledge-intensive NLP yet remains costly: clustering-based methods (i) need careful per-corpus tuning and yield unstable hierarchies, while LLM-based pipelines (ii) depend on expensive task-specific training that does not transfer across domains. We report an empirical finding that bears on both: in Matryoshka Representation Learning (MRL) embeddings, where each leading prefix is itself a usable, coarser vector, prefix length aligns with the coarse-to-fine granularity of gold taxonomies; shuffled- and reverse-dimension controls isolate the alignment to the learned early prefix dimensions. The pattern holds across three architecturally distinct MRL models. To test whether this regularity is usable, we introduce PEEL (Peeling Embeddings to Extract Latent ontologies): a minimal probe over these frozen embeddings, trained once with an order-embedding objective that represents hierarchy as vector containment; a minimum spanning arborescence then decodes taxonomic DAGs. Using only a concept vocabulary (no documents), PEEL recovers taxonomic DAGs with high structural fidelity on two OLLM benchmarks, transferring to a new domain with no target-edge supervision. The end-to-end OLLM pipeline, which jointly discovers concepts and induces structure from document corpora, is reported as a cross-task reference. Pre-trained retrieval embeddings encode a decodable granularity gradient; a minimal probe renders it as explicit parent-child (is-a) structure.