Know Yourself and Know Your Neighbour: A Syntactically Informed Self-Supervised Compositional Sentence Representation Learning Framework using a Recursive Hypernetwork

AI summary

Overview

• Research area: Natural Language Processing, specifically self-supervised learning of sentence representations with explicit syntactic structure.
• Technical level: Advanced. The paper builds on recursive neural networks, Tree-LSTMs, and hypernetworks, and includes the full LSTM-style gating equations for its compositional operators.
• Scope: The paper proposes a recursive hypernetwork that composes sentence representations along a constituency parse tree and trains it with six self-supervised tasks, then evaluates the resulting representations on standard probing tasks against averaging baselines and existing sentence encoders.

Note on completeness: the supplied text is truncated mid-sentence in Section 5. The paper promises an ablation of the six training tasks (Section 5.1) and an analysis of the semantic compositional operator covering robustness to parse tree depth and sensitivity to syntactic information (Section 6), plus conclusions and future work (Section 7). Those results are not present in the available content and are flagged below as not reported.

What This Paper Is About

Most language models produce representations of words or tokens, but many NLP tasks need a single vector for a whole sentence. The common shortcut is to average token embeddings, which discards word order and syntax. The authors' goal is to build a self-supervised method that composes a sentence vector by walking the sentence's constituency parse tree, so that grammatical structure shapes the representation from the start.

Key Contributions

1. A recursive hypernetwork architecture in which two recursive networks operate in parallel: a syntactic network that composes syntactic embeddings with a Tree-LSTM operator, and a semantic network that composes semantic embeddings. The syntactic network acts as a hypernetwork for the semantic network, generating scaling parameters that modulate the semantic compositional operator.
2. Six novel self-supervised training tasks split into syntactic tasks (self category prediction, neighbour category prediction, syntactic position prediction) and semantic tasks (self text generation, neighbour text generation, semantic position prediction), trained jointly with a single aggregate loss whose task weights are learnable.
3. A demonstration that the framework can be initialised from three different off-the-shelf embedding sources (GloVe, RoBERTa, Llama) using a single architecture and dimension setting, with different resulting parameter counts.
4. A probing evaluation on seven tasks reporting accuracy above averaging baselines and several state-of-the-art sentence encoders, plus the claim (details not in the supplied text) that representations are stable across sentence lengths and that semantic composition operators adapt to different syntactic categories.

Main Findings

• Overall probing accuracy: Averaged over the seven non-WC probing tasks, Ours+RoBERTa reaches 0.8189, Ours+Llama 0.8130, and Ours+GloVe 0.8090. The baselines in the same table score 0.7839 (SBERT-WK), 0.7737 (Skip-thought), 0.7633 (Infersent), 0.7620 (average Llama), 0.7592 (average RoBERTa), 0.6534 (SBERT), 0.6450 (average GloVe), and 0.6446 (SimCSE).
• Syntax-sensitive tasks improve most: On TreeDepth, the model scores 0.4933 (Llama), 0.4879 (RoBERTa), and 0.4784 (GloVe), versus 0.4540 for SBERT-WK, the strongest baseline on that task. On TC it scores 0.8756, 0.8712, and 0.8746, versus 0.8277 for Skip-thought and 0.7920 for SBERT-WK.
• Sentence length and semantic tasks: SentLen accuracy is 0.9553 (GloVe), 0.9480 (RoBERTa), and 0.9389 (Llama), against 0.9240 for SBERT-WK. Subj/Obj reach 0.9164/0.8867 with RoBERTa initialisation.
• One reported exception: On CI with Llama-initialised embeddings, the model scores 0.7203, below the Llama average baseline of 0.7493. The authors state this is the only case where the model did not beat its corresponding average baseline.
• Word Content is treated as an outlier: The model scores far lower on WC (0.4436 GloVe, 0.4090 RoBERTa, 0.2541 Llama) than simple baselines (0.8669 for the GloVe average). The authors exclude WC from their analysis, arguing that most recent language model baselines also do poorly there (62% for RoBERTa average, 61% for BERT average, 50% for the BERT CLS token) while bag-of-words approaches do well (80% GloVe average, 98% tf-idf), and that its 1000 target labels inflate the probing classifier and risk overfitting.
• Model size varies with the leaf embedder: With a syntactic embedding dimension of 64, the syntactic compositional operators have 20K parameters. With a semantic embedding dimension of 1024, the semantic compositional operators have 3.7M parameters (GloVe-based), 4.9M (RoBERTa-based), and 13.5M (Llama-based).
• Training cost: On a single A100 GPU, training (excluding parsing and preprocessing) took 19:30 hours for GloVe-based, 20 hours for RoBERTa-based, and 27:45 hours for Llama-based models, over 5 epochs.
• Ablation and composition-operator analyses: The paper states these were performed, but the results are not present in the supplied content.

Methodology in Plain English

The model parses each input sentence into a constituency parse tree, then converts it into a binary tree by adding auxiliary nodes. It then composes representations bottom-up from the leaves (words) to the root (the sentence). Two things are composed at every node: a syntactic embedding using a regular Tree-LSTM operator, and a semantic embedding using a modified Tree-LSTM whose parameters are modulated by the parent's syntactic embedding. This modulation is what makes the syntactic network a hypernetwork for the semantic network, and it lets the model learn different semantic composition behaviour for different syntactic categories rather than reusing one fixed operator everywhere. The root vector is the concatenation of the syntactic and semantic embeddings, so syntax enters the representation both inside the hypernetwork and in the final concatenation.

Leaf embeddings are handled differently per stream: the syntactic leaf embeddings are learned from scratch in an embedding layer, while semantic leaf embeddings come from an off-the-shelf model (GloVe, RoBERTa, or Llama). Out-of-vocabulary GloVe words get a random 300-dimensional embedding; for RoBERTa and Llama, multi-token words are represented by averaging their token embeddings.

Training uses six self-supervised tasks. Three operate on syntactic embeddings (predict the node's own category, predict its sibling's category, predict whether it is a left or right child) and three operate on semantic embeddings (generate the node's own text, generate the sibling's text, predict left/right position). Four tasks are classification tasks served by a small MLP with one 100-dimensional ReLU hidden layer, and two are text generation tasks served by an LSTM auto-regressive decoder. All six losses are combined into one aggregate loss with learnable per-task weights and a regularisation term.

Training data was 100,000 sentences with 2,280,900 tokens from the Wikipedia dump enwiki-20240620-pages-articles-multistream.xml.bz2, parsed with the benepar parser. Training used the Adam optimiser, 5 epochs, a batch size of 8, and an initial learning rate of 0.001 with scheduling.

For evaluation, the authors took the root node representation, trained a separate MLP probe on each of the probing task datasets, and compared against Skip-thought, Infersent, SBERT-WK, sup-simcse-roberta-base, all-mpnet-base-v2, and simple averaged GloVe/RoBERTa/Llama representations. Two of the ten Conneau et al. (2018) probing tasks were dropped (Bigram Shift and Semantic Odd Man Out) because their datasets contain 50% non-fluent samples that would break parsing.

Why This Matters

The paper argues that self-supervised sentence encoders are more scalable than supervised ones (no labelled data such as SNLI needed) and less likely to be biased toward a pretraining objective, but that supervised methods currently win in aggregate partly because self-supervised objectives do not explicitly build in syntax. The framework is a concrete attempt to close that gap by making the parse tree the schedule for composition and by designing objectives that force syntactic information into the embeddings. The reported probing numbers support the claim that this improves the linguistic content of the representation, particularly on syntactically oriented probes like TreeDepth and TC.

Real-world applications:

• Semantic search and retrieval, where sentence-level vectors are indexed and compared, and where word-order-sensitive meaning matters.
• Retrieval-augmented generation and document ranking, where a single representation of a passage is needed without task-specific labelled data.
• Clustering and topic exploration of text corpora, using general-purpose embeddings that are not fitted to one downstream label set.
• Settings with scarce labels, such as low-resource domains or languages, where avoiding supervised data like SNLI is a practical advantage.

Industry relevance: the method is an embedding model that can be dropped into existing pipelines, and the authors note that a specific deployment could fine-tune the framework for a particular downstream task. The costs are relevant too: a training run of roughly 20 to 28 hours on a single A100, plus a dependency on a high-quality constituency parser at inference time.

Future Directions

• Fine-tuning the framework for specific downstream tasks, which the authors explicitly raise as a beneficial use of the framework if task-agnostic generality is not required.
• Completing and reporting the ablation of the six self-supervised tasks to show how much each one contributes, since the paper states this analysis exists but the results are not in the supplied content.
• Extending the composition-operator analysis (robustness to parse tree depth and sensitivity to syntactic categories) to concrete quantitative results, and comparing the operator behaviour across the GloVe, RoBERTa, and Llama initialisations.
• Reducing dependence on valid parse trees, since the authors had to exclude two probing task datasets containing non-fluent sentences, and evaluating on benchmark suites such as MTEB to check transfer beyond probing tasks.
• Investigating inference cost, given that representation generation requires parsing each sentence plus a recursive pass over the tree.

Target Audience

Researchers and graduate students working on sentence embeddings, compositional semantics, or syntax-aware neural architectures; practitioners evaluating self-supervised alternatives to supervised sentence encoders such as Infersent or SBERT-based models; and engineers interested in whether parse-tree-based composition is worth the added parsing dependency and compute in a production embedding pipeline.

Authors’ abstract

Sentence representation learning is still an open challenge in Natural Language Processing. In this work, we propose a new self-supervised framework for learning sentence representations, using a special type of neural network called a recursive hypernetwork. Our proposed model composes the representation of a sentence from representations of words by applying a recursive composition through the parse tree. We maintain a separate syntactic and semantic representation, and the semantic composition is guided by the information from the syntactic representation. To train this model, we introduce a novel set of six self-supervised tasks. By analysing the performance on 7 probing tasks, we validate that the generated sentence representation encodes richer linguistic information than both averaging baselines and state-of-the-art alternatives. Furthermore, we assess the impact of the six proposed self-supervised training tasks through ablation studies. We also demonstrate that the representations generated by our model are stable for sentences of varying length and that the semantic composition operators adapt to different syntactic categories\footnote{Code available: https://github.com/vasudev2020/codi/tree/main}.