FrameEOL: Semantic Frame Induction using Causal Language Models

AI summary

Overview

Research area: Natural Language Processing, specifically semantic frame induction using large language models. Technical level: Intermediate. Scope: This paper introduces FrameEOL, a prompt-based method that uses causal language models to generate frame embeddings for clustering frame-evoking verbs, evaluated on English FrameNet and Japanese FrameNet.

What This Paper Is About

Semantic frame induction is the task of grouping frame-evoking words, usually verbs, into clusters that correspond to the semantic frames they evoke, such as distinguishing the "Finish_Competition" sense of "lost" from the "Losing" sense. Existing high-performing methods rely on masked language models (MLMs) like BERT to build frame embeddings, while causal language models (CLMs) like the GPT and Llama series have not been applied to this task. This paper proposes FrameEOL, a prompt-based approach that adapts a CLM to produce frame embeddings, then clusters them to induce semantic frames.

Key Contributions

1. Introduces FrameEOL, a prompt-based method for obtaining Frame Embeddings that outputs One frame-name as a Label representing a given situation, inspired by PromptEOL and extended for semantic frame induction.
2. Applies in-context learning (ICL) and deep metric learning (DML) to optimize the frame embeddings so they are more suitable for frame induction, with DML implemented using LoRA and triplet loss.
3. Shows that on English FrameNet, the CLM-based method with DML outperforms existing MLM-based methods with DML, achieving BcF scores of 71.9 (Gemma 3, one-step) and 70.9 (Llama 3.1, two-step).
4. Demonstrates on Japanese FrameNet, which has only 3,130 verb examples, that a CLM using only 5 ICL examples achieves performance comparable to an MLM fine-tuned with DML, addressing low-resource settings.

Main Findings

• English FrameNet, DML setting: Applying DML to the CLM-based approach gave the highest performance overall. Gemma 3-12B reached a BcF of 71.9 in one-step clustering and 70.6 in two-step clustering; Llama 3.1-8B reached 70.8 and 70.9 respectively. The best encoder model (RoBERTa large with DML) scored 67.9 in one-step and 69.6 in two-step clustering.
• English FrameNet, ICL setting: Performance improved slightly as demonstrations increased. With 20-shot ICL, Gemma 3 achieved BcF of 59.7 (one-step) and 59.2 (two-step), while Llama 3.1 achieved 58.3 and 56.7. The best MLM without fine-tuning scored 57.6 (two-step), so CLMs surpassed untuned MLMs once at least 20 instances were prepared.
• No benefit from two-step clustering when fine-tuning or using ICL: The paper reports that with DML or ICL, two-step clustering did not improve over one-step clustering, suggesting the learned embedding space already separates frames well.
• Effect of the term "FrameNet" in the prompt: Removing "FrameNet" from the prompt had only a limited effect under ICL and DML settings, indicating that FrameNet information is not directly leaked from the pre-training corpus through that term.
• Japanese FrameNet, DML setting: LLM-jp-3-13B-Instruct3 achieved the highest overall BcF of 61.3 in one-step clustering, surpassing the best MLM-based method's 60.0. In two-step clustering, Llama 3.1 achieved the highest BcF of 60.3, exceeding the MLM-based 58.4.
• Japanese FrameNet, ICL setting: With 5 or more few-shot examples, ICL performance was comparable to MLM fine-tuned with DML. The paper attributes this to the small training set of 1,100 examples for Japanese versus 27,536 for English.
• Unseen frames in test sets: On average 135.3 of 434.3 frames per test set were unseen during training for FrameNet, and 110.7 of 137.0 for Japanese FrameNet.
• Smaller model results: Reported in the appendix, Gemma 3-4B with DML reached BcF 68.2 on FrameNet (one-step), and Llama 3.2-3B with DML reached 69.2 (one-step).
• Training cost: DML training time on a single A100 with 80GB VRAM was 1200 minutes for Gemma 3-12B and 720 minutes for Llama 3.1-8B on English FrameNet; 80 minutes for Gemma 3-12B, 60 minutes for Llama 3.1-8B, and 60 minutes for LLM-jp-3-13B on Japanese FrameNet.

Methodology in Plain English

The researchers designed a prompt template that asks a causal language model to complete a sentence about the frame a verb evokes: The FrameNet frame evoked by "[verb]" in "[sentence]" is. Because CLMs predict the next token, the model's internal representation at the final token ("is") serves as the frame embedding for that verb in context. They then cluster these embeddings to group verbs by the frame they evoke.

To improve embeddings, they add two optimizations. First, in-context learning: they place a few labeled examples (verb, sentence, frame name) before the target instance, testing 5, 10, and 20 demonstrations. Second, deep metric learning: they fine-tune the model with triplet loss so embeddings of the same frame are closer and embeddings of different frames are farther apart, using Euclidean distance of normalized embeddings. Because the models are large, they use LoRA to train only low-rank matrices in linear layers.

Clustering follows prior work: one-step group-average agglomerative clustering, and two-step clustering using X-means for the first step and group-average clustering for the second. Evaluation uses B-cubed Precision, Recall, and F-score (BcF as the primary metric), averaged over three-fold cross-validation and multiplied by 100.

For English, the dataset was extracted from FrameNet 1.7 with verbs as frame-evoking words, averaging 27,537 instances across the three subsets. For Japanese, examples were extracted from Japanese FrameNet, averaging 1,043 instances across the three subsets. Models tested include Gemma 3-12B, Llama 3.1-8B, and LLM-jp-3-13B-Instruct3, compared against BERT base, BERT large, ModernBERT base, ModernBERT large, RoBERTa large, and Japanese BERT and ModernBERT models.

Why This Matters

This work opens a route to semantic frame induction using the same large causal language models that already power many language tasks, rather than relying solely on MLMs. It shows CLMs can match or beat MLM-based methods, particularly for a low-resource language where only a handful of ICL examples are needed.

Real-world applications:

• Automatic construction of frame knowledge bases for languages that lack large manually built resources such as FrameNet.
• Domain-specific frame induction for fields like medicine or cooking, where granularity and concepts differ from general resources.
• Handling emerging concepts, such as new senses of words (the paper cites "streaming" in the sense of watching online videos), by inducing frames without a predefined label set.
• Supporting downstream NLP tasks that depend on understanding situation semantics, such as information extraction and question answering.

Industry relevance: Organizations building multilingual or domain-specific semantic resources can substitute a small number of hand-labeled examples plus ICL for large annotation efforts and model fine-tuning, although the paper notes that large CLMs carry significant computational requirements.

Future Directions

• Testing the approach on languages and corpora beyond English FrameNet and Japanese FrameNet, since the paper states it is unclear how well the method generalizes.
• Reducing the computational requirements of the CLM-based method to make deployment feasible in resource-constrained environments.
• Investigating how many ICL examples are optimal across different languages and data sizes, given the paper's finding that about 20 examples sufficed for Japanese.
• Exploring whether the approach works when frames are entirely novel and no annotated examples exist, given the paper's emphasis on low-resource settings.

Target Audience

Researchers and practitioners in natural language processing, particularly those working on frame semantics, semantic frame induction, representation learning, prompt-based methods, and low-resource language technologies. It is also relevant to engineers interested in applying causal language models with in-context learning or parameter-efficient fine-tuning such as LoRA to structured semantic tasks.

Authors’ abstract

Semantic frame induction is the task of clustering frame-evoking words according to the semantic frames they evoke. In recent years, leveraging embeddings of frame-evoking words that are obtained using masked language models (MLMs) such as BERT has led to high-performance semantic frame induction. Although causal language models (CLMs) such as the GPT and Llama series succeed in a wide range of language comprehension tasks and can engage in dialogue as if they understood frames, they have not yet been applied to semantic frame induction. We propose a new method for semantic frame induction based on CLMs. Specifically, we introduce FrameEOL, a prompt-based method for obtaining Frame Embeddings that outputs One frame-name as a Label representing the given situation. To obtain embeddings more suitable for frame induction, we leverage in-context learning (ICL) and deep metric learning (DML). Frame induction is then performed by clustering the resulting embeddings. Experimental results on the English and Japanese FrameNet datasets demonstrate that the proposed methods outperform existing frame induction methods. In particular, for Japanese, which lacks extensive frame resources, the CLM-based method using only 5 ICL examples achieved comparable performance to the MLM-based method fine-tuned with DML.