Cross-lingual Functional Vectors for Emotion Detection in Large Language Models

AI summary

Overview

Research area: Mechanistic interpretability and multilingual natural language processing — specifically, whether "function vectors" (FVs), latent task representations extracted from LLM activations, can transfer across languages.

Technical level: Intermediate. The paper assumes familiarity with transformer internals (residual streams, attention heads, activation patching), but its central claims are stated in accessible terms.

Scope: A single study evaluating whether FVs extracted from in-context demonstrations in one language can steer multilingual multi-label emotion classification in another language, across two 8B open-source LLMs and five languages.

What This Paper Is About

Function vectors are compact directions in a model's internal activations that appear to encode "what task am I doing," and injecting them can make a model perform that task without demonstrations in the prompt. Prior work tested FVs mostly on simple, structured, monolingual tasks, where their effectiveness was limited on semantically complex tasks like sentiment analysis. This paper asks whether FVs still work on a harder task — multi-label emotion recognition — and, crucially, whether an FV extracted using one language's demonstrations can steer the same task in a completely different language at inference time.

Key Contributions

1. Extends FV evaluation to a semantically complex multilingual setting. The authors show FVs remain effective on multi-label emotion recognition, a task requiring implicit intention understanding and disambiguation of subtle affective expressions, arguing that previously reported limitations are not inherent to FVs.

2. Systematic analysis of FV intervention across transformer layers. Distributing FV injections across multiple layers substantially outperforms single-layer intervention, which the authors offer as a practical strategy for complex semantic tasks.

3. First systematic study of cross-lingual FVs for multilingual semantic classification. The study demonstrates task transfer under both standard clean zero-shot prompts and a controlled perturbed zero-shot setting with distractor labels.

4. Stability analysis of attention-head selection and few-shot complementarity. Each LLM shows a relatively stable range of attention heads for building effective FVs, and that pattern stays consistent across languages. Cross-lingual FVs also consistently improve multilingual few-shot prompting without requiring demonstrations at inference time.

Main Findings

• Clean zero-shot gains are consistent across all languages and both models. On Qwen3-8B with |A| = 20 heads, the no-FV baseline macro-F1 was EN 36.4, DE 16.9, ZH 18.4, ES 41.4, RU 52.5. Every FV source language raised every target: for example EN-sourced FVs gave EN 53.7 ±1.4, DE 44.1 ±0.6, ZH 44.2 ±0.6, ES 62.4 ±1.0, RU 72.1 ±2.8; Russian-sourced FVs gave RU 74.5 ±1.2.

• Perturbed zero-shot results show FVs can induce task behavior without explicit task cues. Without FVs, Qwen3-8B scored EN 0.6, DE 0.5, ZH 0.0, ES 8.0, RU 1.8 — near random in several cases. With FVs, English target rose from 0.6 to 50.2 (Spanish-sourced FVs) and Chinese target rose from 0 to 42.7 (Russian-sourced FVs). On Llama-3.1-8B-Instruct with |A| = 5, the perturbed no-FV baseline was EN 14.1, DE 12.9, ZH 8.4, ES 29.1, RU 35.5, rising to as high as RU 59.0 ±2.1 with Russian-sourced FVs.

• Cross-lingual transfer sometimes beats same-language transfer. In the perturbed setting, cross-lingual FVs outperformed same-language FVs for English, Chinese, and German targets. The authors read this as evidence that FVs capture transferable task-level representations rather than language-specific lexical patterns.

• FV directions from different languages are highly aligned. Pairwise cosine similarity between FVs built from different source languages ranged from 0.94 to 0.96 for Qwen3-8B and 0.85 to 0.93 for Llama-3.1-8B, with small standard deviations across five random seeds. Note that the paper evaluates five of the 28 SemEval languages; results for the remaining 23 are not reported.

• FVs complement, but do not replace, few-shot prompting. With Qwen3-8B standard few-shot, the no-FV baseline was EN 66.6, DE 64.1, ZH 54.1, ES 71.6, RU 73.9. FV-enhanced few-shot improved many configurations — Spanish-sourced FVs gave RU 78.4 ±0.5, which surpassed the no-FV Russian few-shot result of 73.9 and the Russian-sourced FV result of 76.2 ±1.5. The authors characterize this as partial reproduction of the task-steering effect of few-shot ICL.

• Performance plateaus after a small number of heads. In Qwen3-8B, incorporating more informative heads improved and stabilized performance, but gains began to plateau after approximately six heads, suggesting the strongest steering signal is concentrated in a limited subset of heads. The paper does not report a single fixed optimal head count that applies to all models.

• Multi-layer injection matters. When selected heads span only one or two layers, improvements were limited; configurations distributing heads across multiple layers substantially enhanced performance. The exact number of injected layers achieving the best result per configuration is not reported as a single figure.

• Results are stable across demonstration counts. FVs built from randomly drawn 5-shot through 9-shot demonstrations produced consistently improved and stable performance under clean zero-shot prompts on Qwen3-8B.

• Error analysis shows category-specific effects. Aggregated confusion matrices over five languages show that for anger and disgust, true positives and true negatives increased while false positives and false negatives decreased. For joy, true positives rose and false negatives fell, with a small share of other labels misclassified as joy. For fear, sadness, and surprise — described as more subtle and easily confused — false positives dropped substantially and true negatives rose, which the authors interpret as more cautious prediction.

• Small models with injection rival much larger fine-tuned ensembles on quality-per-cost. The authors contrast their inference-time intervention on 8B models with prior SemEval 2025 Task 11 systems PAI (82.3, 73.9, 70.9, 84.8, 88.2) and JNLP (80.4, 69.9, 68.1, 83.0, 89.1), which use ensembles of large fine-tuned LLMs such as Qwen 32B.

Methodology in Plain English

The authors take a model, feed it a handful of labelled examples in one language, and record the internal activations at the last token position. They compare that against a "corrupted" version of the same prompt in which the labels are randomly shuffled, and use the difference — an activation-patching measure called the average indirect effect — to rank which attention heads carry the task signal. The top-k heads by this score form the set used to build the vector. Those heads are identified using Concept_V_Object_5, one of 21 extractive-style tasks from the original FV work, rather than the emotion task itself; the authors note these heads are fixed and shared across tasks.

The function vector itself is the sum of the mean activations of those selected heads on clean emotion-classification prompts, computed separately for each language. At inference, the vector is simply added into the residual stream, following the formulation h_ℓ = h_{ℓ-1} + m_ℓ + a_ℓ + v_{kg}, with the injected layers chosen based on the same indirect-effect scores.

Two prompt conditions frame the test. A clean zero-shot prompt states the task instructions and lists the candidate emotion labels. A perturbed prompt adds distractor categories such as colours and nations alongside the emotion labels, weakening the explicit instruction so that the steering effect of the FV can be isolated. For example, the German perturbed template lists "Colors: yellow, red, blue or none" and "Nation: China, USA, Germany, England or none" before the emotions.

Evaluation uses the SemEval Task 11 dataset across five typologically diverse languages: English (train 2,768 / dev 116 / test 2,767 / total 5,651), German (2,603 / 200 / 2,604 / 5,407), Chinese (2,642 / 200 / 2,642 / 5,484), Spanish (1,996 / 184 / 1,695 / 3,875), and Russian (2,679 / 199 / 1,000 / 3,878). Labels cover joy, sadness, fear, anger, surprise, and disgust, and are multi-label. The metric is macro-averaged F1, following the SemEval protocol. Every experiment is run with five random seeds and reported as mean ± standard deviation. Models are Qwen3-8B and Llama-3.1-8B-Instruct. Zero-shot decoding is greedy on the first generated token; few-shot allows up to 10 new tokens.

Why This Matters

Impact on research. The work pushes back on the earlier finding that FVs fail on semantically complex tasks, showing that a large part of the problem was intervention placement — specifically, using only one layer. It also reframes the cross-lingual debate: rather than asking whether multilingual LLMs have shared language representations in general, it asks whether task-level latent representations transfer, and finds cosine similarities of 0.94–0.96 between FVs built from different languages on Qwen3-8B.

Real-world applications:

• Low-resource language deployment. An FV extracted from English demonstrations can steer emotion classification in a language where labelled demonstrations are scarce or unavailable, avoiding the need to collect and include target-language examples in every prompt.

• Cost and latency reduction in production. Since FVs avoid processing multiple demonstrations per request, they offer a lighter-weight path than few-shot prompting, which the authors frame as beneficial for large-scale practical applications.

• Robustness under adversarial or cluttered prompts. The perturbed setting shows a jump from 0.6 to 50.2 macro-F1 for English when distractors are present, relevant to settings where prompts carry conflicting or noisy instructions.

• Moderation and mental-health-adjacent sentiment screening. Multi-label emotion detection over five languages is directly applicable to flagging anger, fear, or sadness in user-generated content across markets.

Industry relevance. The method requires no parameter updates and works at inference time on 8B-scale open-source models. The authors position this against prior competitive SemEval systems that relied on ensembles of fine-tuned models such as Qwen 32B, noting that their smaller-model intervention "still achieves remarkable performance across languages." The paper also notes the specific finding that a model's optimal head configuration is largely model-specific but consistent across languages, which makes deployment tuning a one-time-per-model exercise rather than a per-language one.

Future Directions

• Label dependency modelling. The limitations section states the study focuses on task-level cross-lingual transfer rather than label dependency, and does not explicitly investigate how FVs capture correlations among co-occurring emotion labels in multi-label prediction.

• Generalization beyond classification. It remains unclear whether the observed cross-lingual transferability extends to other multilingual reasoning or generation tasks, with machine translation named as an example.

• Over-steering in generation. The authors raise the possibility that generation tasks, which involve long output sequences, may suffer if FVs applied throughout the process corrupt the model's latent space and degrade generation quality.

• Systematic layer-selection strategy. The paper establishes that multi-layer injection substantially beats single-layer injection but does not report a single prescribed optimal layer count, leaving open how to select layers principledly for new tasks and models.

Target Audience

Researchers in mechanistic interpretability and multilingual NLP who want to know how far latent task vectors generalize; practitioners building multilingual content classification or moderation systems on 8B open-source LLMs who need efficient alternatives to few-shot prompting; and readers already familiar with the original function vector work (Todd et al., 2024) who want to see it stress-tested on a harder, cross-lingual task. Beginners will follow the high-level argument but will need background in transformer internals to engage with the extraction and injection details.

Authors’ abstract

Function vectors (FVs) have recently emerged as a promising mechanism for steering the behavior of large language models (LLMs) by injecting task-specific latent direction representations derived from in-context demonstrations. While prior studies have shown that FVs can recover task behavior in structured in-context learning settings, their effectiveness on semantically complex tasks and their ability to generalize across languages remain underexplored. We investigate the cross-lingual transferability of FVs using multilingual multi-label emotion recognition as a challenging semantic classification benchmark. Specifically, we examine whether FVs extracted from a source language can steer task behavior in another language under both standard clean and perturbed zero-shot settings without providing demonstrations during inference. Across diverse cross-lingual settings, applying FVs substantially improves performance, suggesting that FVs capture language-agnostic, task-relevant signals rather than purely language-specific lexical patterns, and highlighting their potential as a lightweight and transferable mechanism for multilingual task adaptation. We observe that each LLM exhibits a relatively stable optimal range of attention heads for constructing effective FVs, and the pattern remains consistent across languages. In addition, FVs can partially replicate the task-steering effects of standard few-shot in-context learning while avoiding the computational overhead of processing multiple demonstrations, making them effective for large-scale practical applications. Our code is available at https://github.com/yingjie7/cross_lingual_fvs.