Mapping Semantic & Syntactic Relationships with Geometric Rotation

AI summary

Overview

• Research area: Natural Language Processing / interpretability — specifically the geometry of sentence embedding spaces and the linear representation hypothesis (LRH).
• Technical level: Intermediate (the main text is accessible to anyone familiar with embeddings and cosine similarity; the appendix on Riemannian geometry, geodesics, and Clifford-algebra rotors is Advanced).
• Scope: The paper introduces Rotor-Invariant Shift Estimation (RISE), a method that models discourse-level semantic-syntactic transformations (negation, conditionality, politeness) as rotations on the hypersphere where sentence embeddings live, and evaluates it across 7 languages, 3 embedding models, 3 datasets, and 2 baselines.

What This Paper Is About

Early word embeddings allowed intuitive vector arithmetic ("king" - "man" + "woman" = "queen"), but modern high-dimensional text representations no longer show such straightforward geometric structure, leaving model internals opaque. The authors ask whether sentence-level semantic-syntactic transformations correspond to consistent rotational operations on the curved manifold (unit hypersphere) where normalized embeddings reside. Their goal is to build and test a framework that maps these transformations geometrically and checks whether the mappings transfer across languages and across model architectures.

Key Contributions

1. Introduces RISE (Rotor-Invariant Shift Estimation): a geometric method that canonicalizes a neutral sentence embedding to a fixed reference direction using an orthogonal rotor, computes the Riemannian logarithm to the transformed sentence, averages these into a single "prototype" transformation vector, and applies it to unseen sentences via the Riemannian exponential map.
2. Extends the linear representation hypothesis (LRH) beyond monolingual, word-level settings: the authors state this is the first demonstration that discourse-level semantic-syntactic transformations correspond to consistent geometric operations in multilingual embedding spaces, empirically supporting the LRH at the sentence level — though in curved/geodesic rather than Euclidean form.
3. Demonstrates cross-lingual and cross-model transfer: RISE transformations are shown to operate across languages and across embedding models without reduced performance in some settings, suggesting analogous cross-lingual geometric structure. Cross-model transfer from text-embedding-3-large to bge-m3 follows the statistical mapping approach of Morris et al. (2020).
4. Provides theoretical properties: the appendix shows sequential RISE edits commute (Theorem A.1, stated as first-order commutativity in the theorem heading; the appendix overview describes commutation "up to second order"), and that each transformation can be applied in O(d) time and memory (Proposition A.1).

Main Findings

• Negation is the most robust transformation: it achieves the highest mean rotor alignment score across all model-language combinations, 0.788, with performance ranging from 0.686 to 0.918. It is most geometrically consistent in text-embedding-3-large.
• Conditionality is the most stable: it has the second-highest mean performance at 0.780, with the lowest performance variability across cross-language transfers, 0.038.
• Politeness is the most variable: it ranks third at 0.762 mean, with the highest variability, 0.060, consistent with politeness being culturally and context-dependent.
• Dimensionality does not predict cross-lingual performance: despite its lower dimensionality, bge-m3 (1024-dim) showed the least variance in cross-language performance across all phenomena and languages; text-embedding-3-large (3072-dim) showed the highest cross-language performance; mBERT (768-dim) showed strong monolingual performance but high variability, especially for politeness in cross-language settings.
• Cross-model transfer shows an English-centric bias: transferring from text-embedding-3-large to bge-m3, English achieves 0.80–0.82 similarity across all transformations, other languages cluster around 0.70–0.75, and Zulu consistently scores 0.63–0.66 — a 20% performance gap that persists across conditionality, negation, and politeness.
• Dataset choice changes the picture (Table 2): on BLiMP, scores are 0.929 (text-embedding-3-large), 0.956 (bge-m3), 0.961 (mBERT), average 0.949; on SICK, 0.623, 0.631, and 0.736, average 0.663; on Synthetic Multilingual, 0.771, 0.782, and 0.709, average 0.754. The gap between BLiMP (>0.92) and SICK (0.62–0.74) suggests RISE captures grammatical/syntactic structure better than general semantic relatedness.
• Curved-manifold methods beat rigid linear alignment: RISE and MDV both show strong monolingual performance (RISE 0.97 on BLiMP, 0.84 on SICK; MDV 0.97 and 0.83) and moderate-to-strong cross-language transfer (RISE 0.74–0.89, MDV 0.72–0.91), while Procrustes reaches 0.99 on BLiMP but drops to 0.67 on SICK and 0.25–0.62 cross-language — failing to generalize across languages and models.
• Overall cross-language transfer effectiveness is reported as 77%–95% across typologically diverse languages.

Methodology in Plain English

Normalized sentence embeddings sit on the surface of a unit hypersphere. RISE treats a transformation like negation as a rotation on that sphere. For each pair of sentences (neutral and transformed), the method first rotates the neutral sentence's embedding so that it points at a single fixed reference direction — this puts every pair into a shared frame of reference and controls for what the two sentences already have in common. It then measures the "distance and direction" from the neutral sentence to the transformed one along the sphere's surface (the shortest path, called a geodesic), flattens that path into an ordinary vector using a logarithm map, and averages all these vectors across many sentence pairs into one prototype transformation. To transform a new sentence, it rotates the prototype back into that sentence's frame and walks that distance along the geodesic using an exponential map.

The authors tested this with three embedding models (text-embedding-3-large, 3072-d; bge-m3, 1024-d; mBERT, 768-d), three datasets (BLiMP, SICK, and a synthetically generated Synthetic Multilingual dataset with 1,000 neutral-transformed sentence pairs per language-transformation combination produced by GPT-4.5), and seven languages (English, Spanish, Japanese, Tamil, Thai, Arabic, Zulu), spanning Indo-European, Sino-Tibetan, Dravidian, Afroasiatic, and Niger-Congo families and analytic, agglutinative, and fusional morphological types. Performance is scored as the mean cosine similarity between predicted and target embeddings on held-out test sets, using a published interpretation scale: ≥0.80 strong, 0.65–0.80 moderate, 0.50–0.65 weak, and <0.30 inconsistent or failing. Two baselines were used — Mean Difference Vectors (MDV) and Procrustes alignment — and cross-model experiments used an 80/20 train-test split with PCA-based distributional alignment.

Why This Matters

Impact on research. The paper argues that most steering and probing techniques operate in linear/Euclidean space while modern embeddings live on curved manifolds, and that this geometric mismatch may explain why linear methods generalize poorly. It extends the LRH from word-level, monolingual, categorical concepts to sentence-level, multilingual, discourse-level transformations — while explicitly noting that the linear structures involved are geodesic arcs, not Euclidean lines. It also reports what it calls the first work investigating steering-style methods on embedding models rather than completion models, and on manifolds rather than activations.

Real-world applications (directions the paper's framework supports):

• Interpretability tooling for probing what a model encodes about grammatical and discourse features.
• Steering or controlled manipulation of embedding outputs, since transformations are learned as reusable operations.
• Cross-lingual transfer of learned operations to languages with fewer resources, without retraining per language.
• Auditing multilingual models for bias, given the measured 20% English advantage in cross-model transfer.

Industry relevance. The work comes from TELUS Digital, and its efficiency claim — O(d) time and memory per transformation — matters for production systems where embedding operations must scale. The finding that training methodology and architecture matter more than raw embedding dimensionality has direct implications for teams choosing embedding models for multilingual deployment. The paper also provides a public code repository (https://github.com/fuelix/RISE-steering).

Future Directions

1. Expand beyond three transformations. The authors note their analysis covers only negation, conditionality, and politeness, and call for testing additional semantic and pragmatic phenomena.
2. Validate across more architectures. Although three diverse models were tested, the authors state that additional architectures would strengthen claims about the universality of geometric semantic structure.
3. Reduce reliance on synthetic, English-centric data. The paper flags that using GPT-4.5 for data generation may introduce biases toward English-centric conceptualizations, and recommends more diverse data sources plus validation by native speakers.
4. Address the English-centric bias directly. The authors call for developing more equitable multilingual representations and investigating which language-specific geometric structures are inherent to the models, and caution that claims about universal geometric operations require substantial qualification.

Target Audience

Interpretability researchers studying representation geometry and the linear representation hypothesis; NLP engineers working with multilingual sentence embeddings who need cross-lingual or cross-model transfer of learned operations; and teams building steering, probing, or embedding-manipulation tooling for production systems. Readers who want the practical findings can read the main text alone — a basic grasp of embeddings and cosine similarity suffices — while the appendix's Riemannian geometry, Hadamard-type arguments, and rotor construction are aimed at readers with a stronger mathematical background.

Authors’ abstract

Understanding how language and embedding models encode semantic relationships is fundamental to model interpretability. While early word embeddings exhibited intuitive vector arithmetic (''king'' - ''man'' + ''woman'' = ''queen''), modern high-dimensional text representations lack straightforward interpretable geometric properties. We introduce Rotor-Invariant Shift Estimation (RISE), a geometric approach that represents semantic-syntactic transformations as consistent rotational operations in embedding space, leveraging the manifold structure of modern language representations. RISE operations have the ability to operate across both languages and models without reducing performance, suggesting the existence of analogous cross-lingual geometric structure. We compare and evaluate RISE using two baseline methods, three embedding models, three datasets, and seven morphologically diverse languages in five major language groups. Our results demonstrate that RISE consistently maps discourse-level semantic-syntactic transformations with distinct grammatical features (e.g., negation and conditionality) across languages and models. This work provides the first demonstration that discourse-level semantic-syntactic transformations correspond to consistent geometric operations in multilingual embedding spaces, empirically supporting the linear representation hypothesis at the sentence level.