PromptTailor: Multi-turn Intent-Aligned Prompt Synthesis for Lightweight LLMs

AI summary

Overview

PromptTailor is a system for controllable prompt generation for open-ended text. It expands minimal user instructions into rich, domain-aware prompts while preserving the user’s stated preferences. The system is a quantized 4-bit Llama3-8B model fine-tuned with a lightweight LoRA adapter on 12,300 prompt-refinement dialogues spanning 41 everyday domains, distilled from three stronger LLMs. The adapter attaches to any Llama3-8B base, enabling edge deployment. In human and LLM-judge evaluations across multiple target models and optimization baselines, PromptTailor yields higher preference rates than chain-of-thought prompting and matches or surpasses state-of-the-art prompt optimization methods while requiring fewer model calls (e.g., 3 vs. 9).

What This Paper Is About

Lightweight language models remain attractive for on-device and privacy-sensitive applications, but their responses are highly sensitive to prompt quality. For open-ended generation, non-expert users often lack the knowledge or time to consistently craft high-quality prompts, leading them to rely on prompt optimization tools. A key challenge is ensuring that optimized prompts genuinely align with users’ original intents and preferences.

The paper introduces PromptTailor, an LLM-based prompt generation system that transforms user intents and preferences into enriched and elaborated prompts while maintaining alignment with the original intent. The system generates elaborate, human-readable prompts that are user-friendly and easy to edit. It is evaluated against chain-of-thought prompting, ExpertPrompting, and Evoke, using LLM-based pairwise evaluation and a small human study.

Key Contributions

• PromptTailor: a system for controllable prompt generation for open-ended text that improves model output quality by intent-aligned prompt synthesis.
• A synthetic dataset of 12,300 prompt-refinement dialogues spanning 41 diverse domains, generated using three distinct yet cost-efficient LLMs: GPT-4o-mini, Claude 3 Haiku, and Grok-3-mini. The majority of dialogues (8,200) were generated using Grok-3-mini.
• A “capabilities” mechanism that ensures generated prompts remain intent-aligned while giving the LLM room to improvise. Capability information is gathered from two sources: a list of capabilities deemed necessary to complete the task, and capabilities explicitly mentioned in or inferred from the user’s original intent.
• A quantized 4-bit Llama3-8B model fine-tuned with a lightweight LoRA adapter. The adapter attaches to any Llama3-8B base, enabling edge deployment. It operates as a separate LoRA adapter agent that does not affect other LoRA adapters or components within the model.
• An agent system that automatically executes three model calls to produce an optimized prompt, compared with Evoke’s nine calls.
• Evaluations using an LLM-based evaluator with OpenAI o1-mini as backbone and pairwise comparisons, plus a human study with 10 participants.

Main Findings

Automated assessments used a sample size of 410. Table 1 compared responses from Llama3-8B generated using prompts from the original user intent versus various optimization methods. Each cell shows the number of judgments favoring the response from the original prompt (1), the optimized prompt (2), or indicating no difference (0).

• Original vs. CoT: 177 first better, 216 second better, 17 same.
• Original vs. Expert: 170 first better, 225 second better, 15 same.
• Original vs. Evoke(Llama3): 145 first better, 253 second better, 12 same.
• Original vs. Llama3(FT) [PromptTailor]: 141 first better, 248 second better, 21 same.

Compared to CoT prompting, the fine-tuned model achieved superior performance, with 248 favorable outcomes. Evoke was largely on par with PromptTailor, showing comparable favorable outcomes against original prompts. PromptTailor required three model calls compared with Evoke’s nine.

Cross-model comparisons used four quantized LLMs. Table 2 reports Original vs. Llama3(FT):

• Llama3-8B: 141 first better, 248 second better, 21 same.
• Gemma2-9B: 147 first better, 230 second better, 33 same.
• Mistral-7B: 103 first better, 298 second better, 9 same.
• Llama3-70B: 155 first better, 232 second better, 23 same.

Smaller models such as a quantized 4-bit version of Mistral-7B benefited significantly from optimized prompts, with 298 favorable judgments for optimized prompts versus 103 for original ones. Even with a much stronger target model, Llama-3 70B, prompt optimization remained beneficial. The result demonstrates greater gains for weaker models, with some improvement also seen for stronger ones.

Ablation study results are reported in Table 3, evaluated on responses from a base Llama3-8B model, sample size 410. The prompts for this base model were generated by two separate, fine-tuned optimizers: one built on Llama3-8B and the other on Gemma2-9B. The table compares “Without Cap” versus “Cap” (the full method).

• “Without Cap” vs. “Cap” (Llama3-8B): 171 first better, 212 second better, 27 same.
• “Without Cap”

Authors’ abstract

Lightweight language models remain attractive for on-device and privacy-sensitive applications, but their responses are highly sensitive to prompt quality. For open-ended generation, non-expert users often lack the knowledge or time to consistently craft high-quality prompts, leading them to rely on prompt optimization tools. However, a key challenge is ensuring the optimized prompts genuinely align with users' original intents and preferences. We introduce PromptTailor, a system for controllable prompt generation for open-ended text that improves model output quality by intent-aligned prompt synthesis. PromptTailor expands minimal user instructions into rich, domain-aware prompts while preserving the user's stated preferences. The system is a quantized Llama3-8B model fine-tuned with a lightweight LoRA adapter on 12,300 prompt-refinement dialogues spanning 41 everyday domains, distilled from three stronger LLMs. The adapter attaches to any Llama3-8B base, enabling edge deployment. In human and LLM-judge evaluations across multiple target models and optimization baselines, PromptTailor yields higher preference rates than chain-of-thought prompting and matches or surpasses state-of-the-art prompt optimization methods while requiring fewer model calls (e.g., 3 vs. 9). These results show that a compact student, guided by powerful teachers, can learn effective prompt-generation strategies that enhance response quality while maintaining alignment with user intent.