Concept Tokens: Learning Behavioral Embeddings Through Concept Definitions

AI summary

Overview

• Research area: Natural Language Processing; specifically, representation learning and behavioral control in frozen large language models (LLMs).
• Technical level: Intermediate. Readers should be comfortable with token embeddings, language modeling loss, and prompt-based evaluation, but the core idea is explained conceptually.
• Scope: The paper proposes and evaluates "Concept Tokens" — a single learnable input embedding added to a frozen pretrained LLM and trained only on natural-language definitions of a target concept — across three settings: hallucination reduction in closed-book QA, induction of recasting in second-language teaching, and a qualitative study of a real and a fictional landmark.

What This Paper Is About

The paper addresses how to add a concept to a pretrained LLM and steer its behavior without fine-tuning the model's weights or training on task examples. The authors add one new special token to the vocabulary and optimize only its embedding using multiple definitions of the target concept, with every mention of the concept replaced by that token. The goal is a compact control signal that can be asserted or negated in a prompt to amplify or suppress a behavior associated with the concept.

Key Contributions

1. Introduces Concept Tokens, a method that adds a single new special token to a pretrained LLM and learns only its embedding from a definitional corpus, keeping the entire model frozen and using the standard language-modeling (cross-entropy) objective.
2. Formulates a hypothesis about definition-only supervision: with multiple definitions imposing potentially competing constraints, the best embedding is the one that captures the original concept — in contrast to the single-definition "memory token" setting, where lossless reconstruction of one text is possible.
3. Demonstrates directional behavioral steering in two tasks — hallucination in closed-book QA and recasting in second-language teaching — where negating the concept token suppresses the target behavior and asserting it amplifies it.
4. Provides a qualitative analysis with the Eiffel Tower (a concept already in pretraining) and a fictional "Austral Tower" (a new concept), showing what a learned embedding captures and where it fails.

Main Findings

• Directional effect on hallucination: With Llama 3.1 8B Instruct evaluated on 1000 HotpotQA validation instances, negating the concept token produced 21.90% hallucinated answers versus 28.70% with no instruction, while asserting it raised hallucinations to 31.20% and lowered precision to 34.59 (versus 46.65 with no instruction).
• Lower hallucination did not mean higher precision: Precision (correct answers divided by correct plus hallucinated answers) was nearly constant across the negated interventions and baselines: 44.56 for the negated concept token, 44.80 for the explicit "hallucinations" mention, 44.76 for the definitional corpus in-context, and 46.65 with no instruction. The reduction came mainly from increased abstention (No Answer of 60.50 for the negated token).
• Aggregate similarity but per-instance differences: The negated concept token and the definitional-corpus-in-context baseline produced nearly identical aggregate category proportions, yet per-instance agreement was low (Cohen's κ = 0.35), with each method producing hallucinations where the other answered correctly.
• Directional effect on recasting: When student answers contained errors (n = 215), asserting the concept token produced recasting in 62.33% of cases versus 23.26% with no instruction, while negating it dropped recasting to 20.47% and produced No correction in 71.63%.
• Mentioning the technique is not enough: The "recasting mention" prompt produced recasting in only 16.74% of error cases and explicit correction in 78.14%, suggesting that naming the technique does not reliably induce the intended behavior.
• Trade-off with in-context definitions: Prepending the full definitional corpus achieved the highest recasting rate on erroneous answers (93.95%), but it over-corrected on answers with no errors (90.11% any correction, 82.42% recasting) and asked a follow-up question in only 63.07% of cases, compared to 98.04% for the asserted concept token.
• Concept tokens are more conservative: On the 91 answers with no errors, the asserted concept token produced any correction in 53.85% of cases (40.66% recasting), making it the most conservative among the strategies that promote recasting.
• Real versus fictional concepts: The Eiffel Tower token answered factoid questions correctly (builder and location) and supported jokes, competitor lists, landmark analogies, and accurate summaries. The fictional Austral Tower token captured a coherent theme (a large landmark tower associated with Montevideo) and generalized to relational and creative prompts, but hallucinated specific names, dates, heights, and construction details not present in its synthetic article, and its next-token probabilities ranked Buenos Aires above Montevideo for a location prompt.
• Interpretation: Concept tokens primarily act as a semantic/behavioral attractor — selecting an existing representation when the concept is already in pretraining, or inducing a coherent but factually underdetermined theme when the concept is novel — rather than storing new factual detail.

Methodology in Plain English

The researchers add one new special token to the vocabulary of a pretrained LLM, with its own embedding vector. They collect a set of definitions of a target concept and replace every explicit mention of that concept with the new token, producing what they call an instantiated definitional corpus. They then train only that one embedding vector to minimize the standard next-token prediction loss on this corpus, leaving every other weight in the model untouched. Because several definitions may start differently or impose competing constraints, the embedding cannot simply memorize one text; it must encode a representation that satisfies all definitions as well as possible. At inference, the token is used like any other token in a prompt, and can be asserted ("Generate t_c") or negated ("Do not generate t_c"). All experiments used Llama 3.1 8B Instruct quantized to 4-bit weights with greedy decoding; training hyperparameters are referred to Appendix C and are not detailed in the provided content. Evaluation used an LLM-as-a-judge setup with Gemini 2.5 Flash for the HotpotQA experiment (agreement with human annotations on 100 instances: Cohen's κ = 0.88) and manual annotation for the recasting experiment (Cohen's κ = 0.83 on the first 70 samples for the concept token method).

Why This Matters

Impact on research. The work sits between soft prompting and activation-steering methods: instead of learning a prefix of soft tokens or extracting directions from hidden states, it learns a single input embedding from definitional text alone. It provides evidence that a definition-only signal can shift behavior in a frozen model, and it shows that the method can preserve compliance with other prompt instructions better than pasting a long definition into context.

Real-world applications:

• Educational tutoring systems, particularly for second-language teaching, where recasting can be induced through a compact token rather than a long prompt.
• On-premise and low-resource deployment, since the authors motivate the work by educational settings that benefit from local deployment for data privacy and accessibility, such as rural schools, where smaller models may struggle to follow lengthy instructions.
• Closed-book question answering, where a token can encourage abstention over guessing, though the paper shows this comes with reduced coverage rather than better precision.
• Adding new, previously unknown concepts to a model, as illustrated by the fictional Austral Tower, with the caveat that factual details were unreliable.

Industry relevance. The method requires modifying the input embedding layer and running backpropagation through the full network at training time, which the authors note is computationally expensive. Inference, however, uses a single token rather than a long definition, which can reduce prompt length relative to in-context definitions while better preserving instruction following.

Future Directions

• Investigating how definitional corpus design affects learned behavior: the number and diversity of definitions, the number and placement of concept occurrences, and how precisely the induced behavior matches the conditions described in the definitions.
• Analyzing how concept tokens influence internal activations, potentially combining them with sparse autoencoders to identify which latent features they modulate.
• Exploring composition — whether multiple concept tokens can be combined in a single prompt to steer behavior along multiple dimensions at once.
• Validating the empirical findings across a broader range of model families and scales, since the study was limited by constrained compute.

Target Audience

Researchers and practitioners working on LLM control, parameter-efficient adaptation, and interpretability, as well as applied teams building small, locally deployed educational or tutoring systems. The paper is also useful for readers interested in the limits of definition-only supervision, since it is explicit about where concept tokens succeed (behavioral and semantic steering) and where they fail (reliable storage of new factual details).

Authors’ abstract

We propose Concept Tokens, a lightweight method that adds a new special token to a pretrained LLM and learns only its embedding from multiple natural language definitions of a target concept, where occurrences of the concept are replaced by the new token. The LLM is kept frozen and the embedding is optimized with the standard language-modeling objective. We evaluate Concept Tokens in three settings. First, we study hallucinations in closed-book question answering on HotpotQA and find a directional effect: negating the hallucination token reduces hallucinated answers mainly by increasing abstentions, whereas asserting it increases hallucinations and lowers precision. Second, we induce recasting, a pedagogical feedback strategy for second language teaching, and observe the same directional effect. Moreover, compared to providing the full definitional corpus in-context, concept tokens better preserve compliance with other instructions (e.g., asking follow-up questions). Finally, we include a qualitative study with the Eiffel Tower and a fictional "Austral Tower" to illustrate what information the learned embeddings capture and where their limitations emerge. Overall, Concept Tokens provide a compact control signal learned from definitions that can steer behavior in frozen LLMs.