Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agents

AI summary

Overview

Research area: Natural Language Processing — dialogue systems, conversational AI, and preference-based language model alignment.

Technical level: Intermediate. Readers should be comfortable with standard dialogue-system terminology (task-oriented dialogue, intent detection, supervised fine-tuning) and aware of what preference optimization methods like DPO do, but no specialized mathematics is required to follow the argument.

Scope: The paper introduces TACT, a dataset of dialogues that repeatedly switch between task-oriented and chitchat modes, defines two new metrics for measuring whether an agent can initiate and return from those switches, and shows that fine-tuning plus preference optimization on this data produces agents that manage transitions better than prior datasets and models — including GPT-4o — in both automatic and human evaluation.

What This Paper Is About

Conversational AI has historically been split into two separate worlds: task-oriented systems that help users book flights or find restaurants, and chitchat systems that make small talk. Real conversations do not respect that boundary — a user asking about a hotel booking may pause to chat about a past trip, then return to the booking. Existing datasets and models handle this poorly because they either contain no transitions at all or allow only a single, fixed switch in one direction. This paper builds a dataset and training recipe for agents that can detect when a conversation shifts modes, switch proactively when appropriate, and recover the original task afterward.

Key Contributions

1. TACT dataset (TOD-And-Chitchat Transition). Two new dialogue corpora built on MultiWOZ 2.2 and SLURP containing roughly 7,200 and 9,900 dialogues respectively, averaging about two mode switches and one recovery per conversation, across 11–12 distinct flow patterns (TCT, CTC, TCTCT, TCTC, and others). Previous datasets offered at most one switch and zero recoveries.

2. Switch and Recovery metrics. Two new transition-aware evaluation measures that count, per dialogue, how often an agent attempts to change or restore a dialogue mode, and how often the user accepts that move. This shifts evaluation from static response quality toward flow control over multiple turns.

3. A unified modeling framework. The paper compares in-context learning, supervised fine-tuning (using the FnCTOD function-calling architecture), a modular pipeline, and Direct Preference Optimization (DPO) on a single LLaMA-3.1-8B-Instruct backbone, showing that a DPO-trained single model can replace a multi-component pipeline.

4. A hybrid data-validation pipeline. Dataset quality is checked by combining human-authored evaluation criteria with model-based step-by-step reasoning, addressing the gap that existing methods (G-Eval, Active-Critic) use only one of the two.

Main Findings

• Only TACT-trained agents achieve non-zero transition scores. Models trained on FusedChat, InterfereChat, or plain MultiWOZ produced zero successful mode switches and zero recoveries. The TACT-trained model averaged 1.322 switch attempts with 1.300 successes, and 0.977 recovery attempts with 0.856 successes per dialogue.

• TACT generalizes better than its predecessors. Models trained on FusedChat or InterfereChat performed well on their own test sets but degraded on others, while the TACT-trained model was competitive or better on all task-oriented metrics across the board.

• DPO improves dialogue-level consistency. DPO reached 75.74% joint mode-intent accuracy at the dialogue level — the highest among all methods — despite scoring marginally lower than plain SFT on turn-level intent accuracy, indicating better alignment with the user's overall goal.

• DPO wins on response quality and transition naturalness. Against few-shot GPT-4o, DPO achieved a 40.86% win rate under an LLM judge, and more than doubled the win rate on Interestingness versus SFT. On Transition Naturalness it reached 33.7% versus SFT's 11.3%.

• Human evaluators favored DPO even more strongly than the LLM judge. In a 77-dialogue human study, DPO responses won over 70% of comparisons against GPT-4o, compared with roughly 40% under the LLM judge — suggesting the judge understates the model's real-world advantage.

• Successful recovery does not require returning to the original intent. Break-down analysis found that only about 34% of successful recoveries resumed the same intent as before the interruption; in the rest, the agent appropriately started a new, related intent within the correct mode.

• Transition behavior depends heavily on flow type. The agent was most reliable on TCT flows. On CTC flows it attempted fewer transitions, mirroring the dataset's bias toward user-driven chitchat entries. On TC flows it over-triggered switches, suggesting mode confusion when transition cues are weak.

Methodology in Plain English

The researchers started from two existing corpora and built transitions into them. From MultiWOZ 2.2, which already contains multi-turn task dialogues, they extracted task segments and inserted short chitchat detours at natural boundaries, creating TOD → Chitchat → TOD (TCT) flows. From SLURP, which consists only of single-turn intent utterances, they first expanded single intents into short multi-turn tasks, chained related intents into longer ones, and then wrapped short tasks with small talk at both ends, creating Chitchat → TOD → Chitchat (CTC) flows.

Because this augmentation was partly automatic, they validated the result with a two-part check: human-written scoring criteria supplied the rubric, and a language model (GPT-4o-mini) applied that rubric with explicit reasoning, scoring each dialogue for intent accuracy, transition quality, and naturalness.

For modeling, they fine-tuned LLaMA-3.1-8B-Instruct using the FnCTOD function-calling scheme, which forces the model to predict an intent before generating a response. They then applied DPO on top of this fine-tuned model. To build preference pairs, they generated candidate responses from both their fine-tuned model and GPT-4o-mini, and had Gemini-2.5-Pro judge which was better on sensibleness, specificity, interestingness, and transition naturalness — producing 3,009 preferred/rejected pairs. Evaluation combined standard task metrics, pairwise win rates against GPT-4o rated by both an LLM judge and human annotators, and the two new transition metrics.

Why This Matters

Impact on research. The paper reframes dialogue evaluation away from single-turn response accuracy toward multi-turn flow control. The Switch and Recovery metrics, and the finding that only about a third of recoveries return to the prior intent, give the field a more realistic picture of what "successful" conversation management looks like. The demonstration that DPO can replace a modular pipeline with one model is also a meaningful architectural result.

Real-world applications:

• In-car and voice assistants — the work was supported by Hyundai Motor Company and Kia, and the ability to gracefully handle a driver's digression before returning to a navigation or media request is directly relevant to that setting.
• Customer service and booking agents — a travel or ticket agent that can absorb a customer's anecdote and then bring the conversation back to the outstanding reservation task.
• Technical support bots — handling a user's unrelated aside without losing the diagnostic thread and the state of the problem being solved.
• Companion and social agents — maintaining coherent conversation across long sessions where the user drifts between practical requests and casual talk.

Industry relevance. Production dialogue systems are typically either rigidly scripted or purely reactive. This paper offers evidence that a single fine-tuned, preference-optimized model can be proactive — initiating transitions and steering back to task — without brittle predefined flow graphs, which lowers the engineering cost of adapting to new domains.

Future Directions

• Fine-grained flow analysis. The paper aggregates complex flows such as TCCT and TCTT into broader categories during evaluation, which the authors acknowledge may hide flow-specific behaviors. Disentangling these is an explicit next step.

• Beyond DPO. The authors note it is unclear whether the gains come from DPO specifically or from preference-based learning in general, and propose testing reward modeling or ranking-based fine-tuning as alternatives.

• Reducing evaluation bias. Qualitative results rest partly on a single LLM judge, which may not align perfectly with human preferences even though a human study was run alongside it.

• Real-time flow tracking and preference adaptation. Extending the framework to open-ended agents that must manage dialogue continuity dynamically as the conversation evolves, rather than committing to a fixed flow class.

Target Audience

This paper is most useful to researchers and practitioners working on dialogue systems, conversational agents, and LLM alignment — particularly those interested in unifying task-oriented and open-domain behavior, or in measuring multi-turn conversational quality beyond single-response accuracy. It is also relevant to applied engineers building assistants where users routinely drift off-task, and to dataset designers looking for a template for constructing transition-rich conversational corpora with automated validation.

Authors’ abstract

Conversational agents have traditionally been developed for either task-oriented dialogue (TOD) or open-ended chitchat, with limited progress in unifying the two. Yet, real-world conversations naturally involve fluid transitions between these modes. To address this gap, we introduce TACT (TOD-And-Chitchat Transition), a dataset designed for transition-aware dialogue modeling that incorporates structurally diverse and integrated mode flows. TACT supports both user- and agent-driven mode switches, enabling robust modeling of complex conversational dynamics. To evaluate an agent's ability to initiate and recover from mode transitions, we propose two new metrics -- Switch and Recovery. Models trained on TACT outperform baselines in both intent detection and mode transition handling. Moreover, applying Direct Preference Optimization (DPO) to TACT-trained models yields additional gains, achieving 75.74\% joint mode-intent accuracy and a 70.1\% win rate against GPT-4o in human evaluation. These results demonstrate that pairing structurally diverse data with DPO enhances response quality and transition control, paving the way for more proactive and transition-aware conversational agents.