Don't Shoot The Breeze: Topic Continuity Model Using Nonlinear Naive Bayes With Attention

AI summary

Overview

• Research area: Natural Language Processing, dialogue systems, topic continuity, LLM guardrails, and interpretable conversational AI.
• Technical level: Advanced.
• Scope: This paper proposes an interpretable, attention-enhanced nonlinear Naive Bayes model that detects whether a new sentence in a chatbot conversation remains on topic, especially in long or non-sequential conversations.

What This Paper Is About

LLM chatbots can drift off-topic, which hurts user experience and wastes computation. The paper builds a topic continuity model that estimates whether the current sentence fits the preceding conversation, without the token-length and sentence-attention limits of standard BERT-style Next Sentence Prediction. The goal is an analytical, interpretable formula that works on conversations of any length and handles semantic leaps where a sentence responds to something said many turns earlier.

Key Contributions

1. Reformulates topic continuity as a nonlinear Naive Bayes model with sentence attention and logarithmic nonlinearity, producing an interpretable analytical expression for the probability that a new sentence is on-topic.
2. Introduces an attention functional using max and average pairwise log-probabilities with tanh weighting, so the model trusts a single strong match for normal or leap sentences but falls back to an average when the sentence is likely a topic shift.
3. Adds a residual coefficient based on a sine perturbation around low-confidence predictions, correcting uncertain attention outputs without breaking probability bounds.
4. Validates the model on a privately annotated Amazon customer-service dataset with normal, leap, in-domain topic shift, and out-of-domain topic shift labels, showing strong gains over NSP baselines in long and leap conversations.

Main Findings

• Long-context stability: Standard NSP predictions become unstable after about 300 tokens, while the proposed model remains stable beyond 512 tokens and across much longer conversations.
• Token-gap advantage: For short token gaps, NSP and the proposed model perform similarly. As the gap between the current sentence and its target grows, the proposed model clearly outperforms NSP. For gaps over 512 tokens, accuracy is 0.783 versus 0.637, and F1 is 0.819 versus 0.717.
• Residual term improves uncertainty: On 400 low-confidence examples where the attention term gave probabilities between 0.4 and 0.6, adding the residual term raised precision from 0.55 to 0.62, recall from 0.50 to 0.65, and AUC from 0.47 to 0.61.
• Conversational BERT matters: Among NSP models tested, Conversational BERT performed best, improving AUC from about 68.7% to 82.9% and accuracy from 67.8% to 80.8% over original BERT on the full dataset.
• Interpretable and scalable: The model has linear time complexity in conversation length and can batch attention terms, making it suitable for long conversations where token-limited models fail.

Methodology in Plain English

The researchers start with a simple question: given the previous sentences, is the new sentence still on topic? Instead of feeding the entire conversation into a single BERT model, they break the problem into pairwise comparisons between the current sentence and earlier chunks of the conversation. Each comparison is scored by an NSP model, which estimates whether two text segments belong together.

They then combine these pairwise scores using a Naive Bayes-style formula, but they modify it to avoid the unrealistic independence assumption. A nonlinear attention function decides whether to focus on the strongest pairwise match or average over all matches. If the current sentence clearly responds to one earlier chunk, the strongest match dominates. If the sentence changes topic and has no clear match, the average is used instead.

A residual correction term adds a small adjustment when the attention score is uncertain. This term depends on how topic-specific the current sentence is, estimated by comparing its likelihood under topic-specific data versus general dialogue data. To do this, sentences are encoded with Sentence BERT, and Isolation Forest models produce anomaly scores that are converted into probabilities through cumulative distribution functions. The final result is an interpretable formula that returns a probability that the current sentence continues the topic.

Why This Matters

Research impact: The paper offers a bridge between probabilistic graphical models and modern transformer-based NSP, showing that an interpretable analytical formula can outperform token-limited models in long and non-sequential conversations. It also introduces a rare dataset with leap-conversation annotations, where the specific earlier sentence being responded to is labeled.

Real-world applications:

• Customer service chatbots that must stay focused on product, billing, or support topics.
• LLM guardrails that detect off-topic or in-domain topic shifts and redirect the conversation.
• Long-running assistants where conversations exceed standard transformer context limits.
• Computational resource management, since off-topic conversations can be ended or rerouted early.

Industry relevance: The method is designed for production business scenarios, especially Amazon-style customer service. It supports per-product-line probability models stored in cache, linear-time processing, and responsible, interpretable LLM deployment without relying solely on black-box classifiers.

Future Directions

• Release the planned public dataset of LLM-to-LLM conversations on topics such as machine learning and report evaluation results on it.
• Test the model on real user conversations, multiple domains, languages, and product lines beyond online video streaming.
• Integrate the topic continuity score directly into LLM decoding or guardrail loops for real-time topic steering and conversation repair.
• Improve the calibration of the OOD-based residual term and compare against newer long-context transformers and LLM-as-judge approaches.
• Extend the framework to adversarial topic shifts, prompt injection, and multi-party conversations.

Target Audience

This paper is most useful for NLP researchers, dialogue system engineers, and machine learning practitioners building LLM guardrails or customer-service chatbots. It also benefits interpretability and safety researchers interested in probabilistic dialogue modeling, and product teams that need scalable topic continuity checks for long conversations.

Authors’ abstract

Utilizing Large Language Models (LLM) as chatbots in diverse business scenarios often presents the challenge of maintaining topic continuity. Abrupt shifts in topics can lead to poor user experiences and inefficient utilization of computational resources. In this paper, we present a topic continuity model aimed at assessing whether a response aligns with the initial conversation topic. Our model is built upon the expansion of the corresponding natural language understanding (NLU) model into quantifiable terms using a Naive Bayes approach. Subsequently, we have introduced an attention mechanism and logarithmic nonlinearity to enhance its capability to capture topic continuity. This approach allows us to convert the NLU model into an interpretable analytical formula. In contrast to many NLU models constrained by token limits, our proposed model can seamlessly handle conversations of any length with linear time complexity. Furthermore, the attention mechanism significantly improves the model's ability to identify topic continuity in complex conversations. According to our experiments, our model consistently outperforms traditional methods, particularly in handling lengthy and intricate conversations. This unique capability offers us an opportunity to ensure the responsible and interpretable use of LLMs.