C$^3$TG: Conflict-aware, Composite, and Collaborative Controlled Text Generation

AI summary

Overview

Research area: Natural Language Processing — specifically Controlled Text Generation (CTG) with large language models.

Technical level: Advanced. The paper assumes familiarity with KL divergence, energy-based optimization, autoregressive decoding distributions, and parameter-efficient fine-tuning (LoRA).

Scope: The paper proposes C$^3$TG, a two-phase framework that steers a large language model toward multiple user-specified attributes at once (emotion, style, tone, topic, toxicity) while resolving conflicts between those attributes and preserving fluency.

What This Paper Is About

Existing controlled text generation methods generally toggle one simple attribute at a time, and when several desired attributes clash they have no mechanism to coordinate them — improving one attribute can suppress or distort another. They also lack any feedback loop that lets the output be iteratively corrected after it is generated. C$^3$TG attacks both gaps by pairing a generator LLM with lightweight BERT classifiers across 17 attribute subcategories, fusing attribute distributions during decoding and then refining the text through an energy-driven, agent-guided rewriting loop.

Key Contributions

1. A collaborative LLM-plus-small-evaluator design. C$^3$TG pairs an LLM with specialized BERT classifiers across 17 subcategories spanning emotion, style, tone, and topic (plus a toxicity dimension), expanding controllable attributes while also suppressing toxic output. No architectural changes or expensive retraining of the base model are required.

2. A weighted KL-divergence fusion rule for generation. During decoding, the framework minimizes a weighted KL divergence between the sampling distribution and the per-attribute prior distributions, which the authors prove has a closed-form optimum: the weighted geometric mean of the attribute priors.

3. A composite energy function with conflict penalties. The optimization phase scores the text with classifier deviations from target intensities plus penalty terms that constrain drift in non-primary dimensions, and a zero-shot Llama2-7B Feedback Agent turns those scores into rewriting prompts.

4. Empirical validation across two story datasets. Evaluations on ROCStories and WritingPrompts cover automated metrics and human evaluation, plus dedicated "conflict" and "overlap" experiments and an ablation study.

Main Findings

• Attribute accuracy beats baselines. On ROCStories, C$^3$TG reaches 90.39 attribute accuracy versus 89.45 for LLM-based Prompt, 87.53 for Model Arithmetic, and 79.03 for LLM-based Fine-tuning. On WritingPrompts it reaches 85.56 versus 84.23 (Model Arithmetic), 80.02 (LLM-based Prompt), and 75.00 (LLM-based Fine-tuning).

• Fluency improves at the same time. Perplexity is 4.04 on ROC and 3.68 on WP for C$^3$TG, versus 5.37 / 9.65 for LLM-based Prompt and 11.08 / 14.30 for Model Arithmetic. Lower is better.

• Diversity is highest among compared systems. Distinct-1/2/3 scores are 0.53/0.74/0.90 on ROC and 0.47/0.55/0.84 on WP; the next closest is LLM-based Prompt at 0.47/0.71/0.89 and 0.42/0.50/0.82.

• Toxicity is lowest. Table 1 reports a toxicity value of 0.12 for C$^3$TG, against 0.16 for Model Arithmetic and LLM-based Fine-tuning, 0.29 for LLM-based Prompt, and 0.39 for PPLM. In the ablation table, the full system is reported at 0.12 (ROC) and 0.24 (WP).

• Human evaluation agrees. Five independent domain experts rated on a 5-point Likert scale. C$^3$TG scored 4.74 attribute alignment, 4.53 fluency, 4.45 diversity on ROC, and 3.65, 3.88, 4.05 on WP.

• Conflict and overlap handling is stronger. Tested on 30% of ROCStories with a negative pair ("fear 0.7 vs. joy 1.0") and a positive pair ("romance 0.7 + love 0.7"), C$^3$TG achieved average absolute bias of 0.08 (conflict) and 0.07 (overlap), versus 0.19 and 0.12 for LLM-based Prompt and 0.27 and 0.22 for Model Arithmetic. Perplexity was 4.54 and 4.13, and drift 0.16 and 0.18.

• Every component matters. Ablation: generation-only yields 65.22 / 62.17 accuracy, optimization-only 59.40 / 55.08, and removing the overlap penalty 78.46 / 73.56, versus 90.39 / 85.56 for the full system. More iterations help: 1-iteration gives 74.21 / 70.09 accuracy, 2-iteration 85.62 / 81.13, and the full three-stage run 90.39 / 85.56.

• Runtime cost is modest. C$^3$TG is 1.6x slower than the fastest baseline while delivering roughly 3% higher attribute accuracy and 25% lower toxicity. Energy-based early stopping cuts end-to-end latency by 38–42% and removes 2.4 rewrite iterations per sample on average.

Methodology in Plain English

Phase 1 — Generation. The base generator is Llama2. In parallel, the authors fine-tune separate Llama2 models on attribute-specific corpora, so each one produces a "prior" distribution over the next token for its attribute. The user supplies importance scores (for example, Joy 0.9, Polite 0.8). The framework asks: what single next-token distribution sits closest to all of these attribute distributions at once, weighted by their importance? Using Lagrange multipliers, the authors show the answer is the weighted geometric mean of the attribute priors, normalized so the probabilities sum to one. Tokens are sampled from that combined distribution.

Phase 2 — Optimization. A single pass rarely satisfies conflicting or interdependent attributes, so the text is refined. BERT classifiers score the text on each dimension; the deviation from the target, weighted by user importance, forms the first half of an energy function. The second half penalizes movement in all the dimensions that are not currently being optimized, so fixing one attribute doesn't quietly wreck the others. Penalty coefficients are set according to experimentally derived correlations between attributes.

The Feedback Agent. A zero-shot Llama2-7B agent inspects these scores and writes rewriting prompts. It runs a three-stage chain: (1) Core Attribute Calibration, targeting the largest deviations; (2) Attribute Balancing Adjustment, using intensity modifiers like "slightly" or "significantly" and explicitly naming attributes to hold steady; (3) Global Fine-tuning, a consolidated final polish. After each rewrite it computes the change in energy; it stops when energy drops and falls at or below the convergence threshold of 0.025. If the maximum iteration count is reached first, it returns the lowest-energy text.

Setup details. Llama2-7B was fine-tuned with LoRA (rank 8, alpha 16, dropout 0.1), AdamW with weight decay 0.01 and learning rate 2×10⁻⁵, linear warmup over 1,000 steps then cosine decay, 5 epochs, batch size 32, on NVIDIA A100 GPUs, seed 42; trainable parameters are about 0.3% of the base model. BERT classifiers used AdamW with weight decay 0.01, learning rate 3×10⁻⁵, batch size 32, 3 epochs, seed 42. Evaluation used ROCStories (98,162 stories, 53.5 average tokens) and WritingPrompts (303,358 examples, 675–735 tokens).

Why This Matters

Impact on research. The paper reframes multi-attribute control as a conflict-resolution problem rather than a single-attribute steering problem. It supplies a closed-form decoding rule (the weighted geometric mean of attribute priors) and shows that adding an explicit penalty for non-target dimensions meaningfully reduces the collateral damage that plagues previous decoding-intervention methods. It also introduces iterative, feedback-driven refinement into a pipeline where the earlier state of the art was single-pass.

Real-world applications:

• Content moderation and rewriting — the /pol/ dataset experiment demos controlled rewriting toward non-toxic text while keeping the original meaning.
• Marketing and brand voice — enforcing simultaneous constraints such as tone: professional, style: humor, topic: courage.
• Educational and children's content — combining a value-oriented topic (justice, humanity, courage) with a controlled emotional register.
• Assistive writing tools — letting a user dial attribute intensities up or down and see real-time attribute estimates as text is produced.

Industry relevance. The framework needs no architectural modification and no retraining of the base model, and the trainable adapter accounts for roughly 0.3% of base parameters. That makes it a plug-in layer that can sit on top of an existing LLM deployment, which matters for teams that cannot afford full fine-tuning. The 1.6x latency overhead and 38–42% latency savings from early stopping are the kinds of numbers that decide whether such a layer is deployable.

Future Directions

• Scaling the attribute inventory. The current system covers 17 subcategories plus toxicity; whether the same weighted-geometric-mean decoding stays stable with many more simultaneously active dimensions is untested.

• Removing the dependency on labeled classifier data. The approach depends on BERT classifiers trained on labeled corpora (Social Network Sentiment, xSLUE, Domain Q&A/Instructions, Toxicity Review). Classifier quality bounds system quality, and no analysis of classifier error propagation is reported.

• Latency and iteration budget. The system is 1.6x slower than the fastest baseline and the convergence threshold is fixed at 0.025. Adaptive thresholds or better early stopping could narrow the gap further; the paper does not report how often the maximum iteration limit is hit instead of true convergence.

• Generalization beyond English narrative text. All experiments use two English story corpora (ROCStories and WritingPrompts). Behavior on dialogue, code, long-form documents, or non-English text is not reported.

Target Audience

Researchers and engineers working on controllable text generation, LLM alignment, and inference-time steering. It is most useful for readers already comfortable with decoding-time interventions (PPLM-style methods, energy-based control, weighted KL fusion) and for practitioners who need multi-attribute control without retraining a base model. Readers looking for an introductory treatment of CTG would need to consult the cited background work first, and readers focused on parameter-efficient fine-tuning alone will find only the adapter configuration relevant here.

Authors’ abstract

Recent advancements in large language models (LLMs) have demonstrated remarkable text generation capabilities. However, controlling specific attributes of generated text remains challenging without architectural modifications or extensive fine-tuning. Current methods typically toggle a single, basic attribute but struggle with precise multi-attribute control. In scenarios where attribute requirements conflict, existing methods lack coordination mechanisms, causing interference between desired attributes. Furthermore, these methods fail to incorporate iterative optimization processes in the controlled generation pipeline. To address these limitations, we propose Conflict-aware, Composite, and Collaborative Controlled Text Generation (C$^3$TG), a two-phase framework for fine-grained, multi-dimensional text attribute control. During generation, C$^3$TG selectively pairs the LLM with the required attribute classifiers from the 17 available dimensions and employs weighted KL-divergence to adjust token probabilities. The optimization phase then leverages an energy function combining classifier scores and penalty terms to resolve attribute conflicts through iterative feedback, enabling precise control over multiple dimensions simultaneously while preserving natural text flow. Experiments show that C$^3$TG significantly outperforms baselines across multiple metrics including attribute accuracy, linguistic fluency, and output diversity, while simultaneously reducing toxicity. These results establish C$^3$TG as an effective and flexible solution for multi-dimensional text attribute control that requires no costly model modifications.