Introducing Axlerod: An LLM-based Chatbot for Assisting Independent Insurance Agents

AI summary

Overview

Research area: Applied natural language processing — LLM-based conversational agents (chatbots) in the insurance/insurtech domain, specifically agent-assistive rather than customer-facing systems.

Technical level: Intermediate. Readers need some familiarity with large language models, retrieval-augmented generation (RAG), tool calling, and web application architecture, but the paper is primarily a system description and empirical evaluation rather than a novel algorithmic contribution.

Scope: One sentence — the paper describes the design, implementation, and internal evaluation of Axlerod, an LLM chatbot built with Safety Insurance that helps independent insurance agents retrieve policy, coverage, and documentation information, reporting 93.18% overall accuracy and a 2.42-second average reduction in search time.

What This Paper Is About

Insurance agents spend significant time locating policy details, coverage information, and documentation scattered across internal systems. While most AI chatbot research in insurance targets customers, this paper argues that agent-facing assistants are a safer and more immediately useful application, because agents have the domain expertise to validate outputs and detect hallucinated responses. The goal is to build a working prototype (Axlerod) that answers agents' policy and coverage questions by querying live internal databases, and to measure how accurate and how much faster it makes their work.

Key Contributions

1. First exploration of agent-assistive LLM chatbots in insurance. The authors state this is the first study to examine LLM-based conversational agents as support tools for insurance agents (rather than policyholders), aimed at improving agents' ability to interact with customers.
2. A fully operational end-to-end system. Axlerod queries a multi-source insurance data stack spanning hundreds of thousands of live policies and associated documents, connecting an LLM to Safety Insurance's internal systems.
3. Empirical evidence on LLM tool use in structured insurance tasks. The paper quantifies accuracy and robustness across four task types using 1,200 queries, informing future applications and benchmarks in the domain.
4. Smoltalk, a lightweight agentic microframework. Described in the system design (though not listed in the formal contribution list), Smoltalk handles LLM–tool interactions with an OpenAI-compatible API that embeds in FastAPI applications, developed to avoid the complexity of third-party frameworks.

Main Findings

• Overall accuracy of 93.18%. Axlerod correctly answered 1,118 out of 1,200 queries across four task types.
• Per-task accuracy varies substantially. Bill plan identification reached 99.3%, AutoPay enrollment 99.0%, and covered-vehicle identification by auto policy number 93.7%. Correctly identifying policy numbers was lowest at 80.7%.
• Policy number errors traced to name ambiguity. The majority of the 80.7% policy-number errors were attributed to ambiguities arising from common or frequently occurring names. By design, Axlerod asks the user for additional input when an initial query returns more than five results.
• Average search time fell from 7.55 to 5.13 seconds. Across two agents completing four tasks, the average reduction was 2.42 seconds.
• Time savings depend on task complexity. Find Policy Number went from 7.6 to 5.3 seconds, Get Autopay Eligibility from 5.4 to 5.2 seconds, Get Covered Vehicles from 14.0 to 5.1 seconds, and Get Bill Plan from 5.3 to 4.9 seconds. The paper reports minimal or no time advantage for simple tasks when the user already has the policy number, but notably faster retrieval for complex tasks requiring navigation across multiple screens or name-based search.
• Benefit is largest for unfamiliar users. Compared with users unfamiliar with the application's search functionality and page layout, Axlerod typically retrieved information several seconds faster.
• Average cost of $0.0075 per answer. Calculated from costs reported by LiteLLM during testing; the authors note the effective cost is significantly lower given agent efficiency gains.
• Model migration. The system initially used the 80 billion-parameter Llama 3.1 model (open-weight, runnable on internal hardware) and transitioned to Google's Gemini 2.5 Pro, motivated by its capabilities and alignment with the company's existing Google Cloud adoption.

Methodology in Plain English

Axlerod is a lightweight application that sits between an LLM hosted on Google's cloud infrastructure and Safety Insurance's internal databases. It exposes three functions to the model as "tools":

• policy_detail — retrieves comprehensive details for a given policy number.
• policy_search — queries a Typesense search engine to find a policy by policyholder name or address.
• documentation_search — searches and accesses policy documentation.

When an agent submits a prompt, the application forwards it to the LLM along with structured instructions and metadata about the available tools. The LLM either answers directly or issues a structured tool call; the application executes the function, retrieves the data, returns it to the LLM, and the LLM generates a refined response for the interface. The chatbot is embedded in the agent-facing web application, so it knows which policy the agent is currently viewing and can interpret references like "this policy."

The data behind these tools includes a policy database of about 730,000 personal and commercial policies (policy number, policy type, policyholder name and address, effective and termination dates, billing dates, installment plans, upcoming payment amounts), coverage and claims information (limits, vehicles covered, claim amounts and descriptions), and a documentation database of roughly 400 MB of largely unstructured company and industry documents specific to state and policy type.

The technology stack includes Typesense for search, Python 3.11 with httpx for API interaction, FastAPI for the web API, Smoltalk for LLM/tool orchestration, LiteLLM as an LLM gateway translating OpenAI-format requests to Google's API with Vertex AI authentication, Open WebUI during early development, and the Tampermonkey userscript manager for prototyping the embedded chat interface.

Evaluation was conducted in a controlled internal environment using scripted and ad-hoc queries representative of agent workflows. Testing combined manual testing through the integrated chat widget with automated tests using Python's unittest module, sending questions directly to Axlerod's API and comparing against known truth. Test policies were chosen at random from the current active policy database. Query types included due date and amount due, policyholder's auto policy number, covered vehicles, and bill plan — each asked in multiple variations representing different language styles and registers. A separate time study assigned two users to complete four tasks with the chatbot and without it.

Why This Matters

Impact on research. The paper reframes insurance chatbots from a consumer-facing to an agent-facing problem, arguing that domain experts are better positioned than consumers to validate output accuracy and detect hallucinated responses. It provides quantified accuracy and latency evidence for LLM tool use in a structured, regulated workflow, and introduces a deliberately minimal alternative to heavier agentic frameworks.

Real-world applications:

• Faster policy and coverage lookup during live client calls, reducing the time an agent spends navigating screens while a customer waits.
• Vehicle and coverage verification using an auto policy number, where Axlerod cut average task time from 14.0 to 5.1 seconds.
• Billing question handling, including due dates, amounts due, bill plans, and AutoPay enrollment status.
• Onboarding and support for less experienced agents, since gains were largest for users unfamiliar with the application's search functionality and page layout.

Industry relevance. The US property and casualty market described in the paper includes more than 4,100 carriers with combined revenue over USD 1 trillion, and the US insurtech sector comprises approximately 1,500 companies with revenues exceeding $9.2 billion in 2024. The paper positions Axlerod explicitly as an augmentation layer rather than a replacement, keeping humans in the loop — a framing consistent with the National Association of Insurance Commissioners' 2020 Principles on Artificial Intelligence, which emphasize fairness, ethics, transparency, accountability, compliance, and security. The authors note that carriers must also comply with evolving state legislative frameworks, creating an imperative for careful system design and compliance monitoring.

Future Directions

• Improve accuracy and mitigate fragmented data. Reducing the impact of difficult-to-access internal data sources is named as a priority, alongside generally improving accuracy — the 80.7% policy-number identification rate suggests one concrete target.
• Refine the interface and integrate with PowerDesk. Smoother embedding within the agents' existing workflow application is a stated goal.
• Expand validation with practicing agents. Broader user studies and more complex multi-step tasks are planned to assess real-world utility, agent trust, and efficiency in both human-in-the-loop and fully automated workflows.
• Benchmark Smoltalk against established frameworks. The authors intend to position their lightweight framework against LangChain and Microsoft's Semantic Kernel, clarifying its advantages in simplicity and ease of integration.

Target Audience

Applied NLP and LLM engineers building tool-using agents; insurtech product and platform teams evaluating LLM deployment in regulated workflows; carrier IT and operations leadership weighing agent-assistive AI; researchers studying trust, oversight, and human-in-the-loop design in high-stakes domains; and graduate students looking for a concrete end-to-end case study that pairs system architecture with quantitative evaluation.

Authors’ abstract

The insurance industry is undergoing a paradigm shift through the adoption of artificial intelligence (AI) technologies, particularly in the realm of intelligent conversational agents. Chatbots have evolved into sophisticated AI-driven systems capable of automating complex workflows, including policy recommendation and claims triage, while simultaneously enabling dynamic, context-aware user engagement. This paper presents the design, implementation, and empirical evaluation of Axlerod, an AI-powered conversational interface designed to improve the operational efficiency of independent insurance agents. Leveraging natural language processing (NLP), retrieval-augmented generation (RAG), and domain-specific knowledge integration, Axlerod demonstrates robust capabilities in parsing user intent, accessing structured policy databases, and delivering real-time, contextually relevant responses. Experimental results underscore Axlerod's effectiveness, achieving an overall accuracy of 93.18% in policy retrieval tasks while reducing the average search time by 2.42 seconds. This work contributes to the growing body of research on enterprise-grade AI applications in insurtech, with a particular focus on agent-assistive rather than consumer-facing architectures.