Medication counseling with large language models: balancing flexibility and rigidity

AI summary

Overview

• Research area: Human-Computer Interaction, applied large language model (LLM) systems, and healthcare/pharmacy conversational agents. The work sits at the intersection of multi-agent LLM design, retrieval augmented generation (RAG), and medication safety.
• Technical level: Intermediate. The paper assumes familiarity with LLM agents, tool calls, ReAct reasoning, context windows and hallucination, but presents no formal mathematics or quantitative benchmarks.
• Scope (one sentence): The paper describes a prototype multi-agent LLM system that provides counseling about emergency contraceptive pills (ECPs), focusing on how to balance the flexibility of LLMs against the rigidity required when errors are costly.

What This Paper Is About

LLM-based agents can converse far more naturally than rule-based chatbots, but in a pharmacy context that flexibility is dangerous because LLMs hallucinate, while the opposite extreme — an overly rigid rule-based system — also fails when a customer describes a situation the rules did not anticipate. Most prior LLM pharmacy work covers many medications in short interactions; this paper deliberately does the reverse, studying one medication (ECPs) over a longer, more structured conversation that more closely resembles counseling by a human pharmacist. The goal is a working prototype that stays faithful to a legally mandated counseling procedure while still adapting to how real customers actually describe their conditions.

Key Contributions

1. A proof-of-concept multi-agent system for ECP counseling. The system is a multi-agent system (MAS) built from three ReAct agents — the Conversationalist, the Medicine interpreter, and the Symptom assessor — implemented as a graph where agents are nodes and interaction paths are edges (mandatory paths solid, optional paths dashed).

2. A conversation specification derived from a hierarchical task analysis. The counseling procedure was decomposed into 35 goals and sub-goals, with the five illustrative goals being: (1) check time since intercourse, (2) check for contraindications, (3) present side effects and the underlying mechanism of the pills, (4) offer choices, and (5) offer advice. The specification was converted into YAML because its rigid token structure and hierarchical organization are easily interpreted by LLMs. The paper also states that the authors developed an extension of the hierarchical task analysis method for more fine-grained analysis.

3. A set of design techniques to increase determinism without removing conversational flexibility. These include tool use, RAG-style retrieval of contraindications, task decomposition across agents, few-shot prompts, follow-up (context-refining) questions, similarity-matching tools with tunable thresholds, a dedicated re-labeling tool, and giving the customer-facing agent a name to make instructions less ambiguous.

4. A proposed evaluation approach and a discussion of ethical/legal issues. The paper argues that standard natural language generation metrics are inadequate for this kind of system and proposes human interaction evaluations (HIE) as an alternative, alongside a discussion of transparency, accountability, explainability and liability.

Main Findings

• Task decomposition was forced by agent overload. The Medicine interpreter was originally responsible for both classifying/clarifying conditions and identifying viable ECPs, which overburdened it and produced a drop in response quality. The work was therefore split: the Symptom assessor assesses symptoms and compiles a list of contraindications, and the Medicine interpreter uses that list to find ECPs that are safe for the customer.

• Pure inference for extracting contraindication terms failed reliably. When terms were extracted purely by inference and used as tool arguments, the system mishandled cases with several similar matches (for example potato or corn starch), sometimes choosing the first match, sometimes no term at all, and sometimes re-using the original input term as the argument. The tools find_most_similar_word_regular_medications_and_diseases and find_most_similar_word_allergies were introduced to return the most similar word or words from their respective tables.

• Similarity thresholds act as a safety margin. The find_most_similar_word_* tools have a threshold parameter; the Symptom assessor calls them with the default value of 0.6, while the Medicine interpreter calls them with 0.8 so that terms remain correct even after a small error such as "astma" being input instead of "asthma."

• Tool calls keep the context window clean. Because all contraindications can be accessed through a tool call, the information does not pollute the LLM's context when it is not needed — described in the paper as a form of retrieval augmented generation.

• Some tasks were moved to their own tool to reduce agent burden. The Symptom assessor struggled when asked to handle everything from deciding whether a disease needed re-labeling to forming the final list of relevant terms. Re-labeling was therefore moved into classify_contraindication, a simple call of the form "if the input term X is a type of Y, then output Y else None" — for example, input "Celiac disease" produces output "Severe Malabsorption Disorder."

• Customer language does not match the knowledge base. The contraindications table includes disease categories such as "Severe Malabsorption Disorder (e.g., Crohn's)" and conditional contraindications such as "Asthma (if glucocorticoids)." The paper states it is unlikely a customer would describe asthma as "Asthma, for which I take medication containing glucocorticoids," or describe celiac or Crohn's disease as "Severe Malabsorption Disorder." Handling this is presented as a key reason the system needs LLM flexibility, because pre-defined states cannot capture that complexity.

• LLMs rarely admit uncertainty. Hallucinations are described as particularly problematic because LLMs rarely admit to lacking knowledge or being wrong and instead produce confidently incorrect responses. This behavior was prominent when discrepancies arose between the specification and the actual conversation, potentially causing a step to be skipped.

• Standard evaluation metrics were judged unsuitable. BERTscore favors token-level similarity and does not consider fluency, coherence or factuality; BARTscore suffers from insensitivities and biases on challenging texts; and both are static, single-turn evaluations rather than multi-turn conversation evaluations.

• No quantitative results are reported. The paper reports a prototype and its design rationale; it does not report benchmark scores, participant counts, or measured accuracy for the system.

Methodology in Plain English

The authors began from a legally mandated counseling procedure for ECPs in Finland. They broke that procedure down using a hierarchical task analysis (and an extension of it) into 35 goals and sub-goals, then wrote the required content of the conversation into a YAML specification that an LLM can follow, alongside a knowledge base describing ECP composition and contraindications. Together, the specification and knowledge base define success: an interaction succeeds if all required topics were discussed and the customer received the right set of ECPs.

The conversational system itself is a graph of three agents, all of which are ReAct agents — meaning they reason and then act, for example by calling an external function. The Conversationalist talks to the customer, follows the YAML procedure, asks follow-up questions when answers are unclear, and finally tells the customer which ECPs, if any, they can take. It deliberately knows little about the other agents — only what to pass in, what each agent is for, and what to expect back — so that its context window stays focused on its core tasks. The Symptom assessor identifies, matches and re-labels terms, asks follow-up questions (including about contraindications customers might not think of, such as breastfeeding), and compiles the list of relevant contraindications. The Medicine interpreter takes that cleaned list and uses the tools check_pill_contraindicating_allergies and check_pill_contraindicating_medications_and_diseases to return the ECPs that are safe.

Tools are used to keep the system rigid where rigidity is warranted. For example, if a customer reports lactose intolerance and some ECPs contain lactose monohydrate, the agent should ask whether the intolerance is severe and, if so, treat those ECPs as unsuitable. Follow-up questions are supported by giving the LLM a tool that returns a response it can act on, following the pattern "if the tool returns X, ask the customer Y," and few-shot prompts are used to teach confirmation and re-confirmation behavior, inter-agent communication, and tool usage.

Why This Matters

• Impact on research: The paper flips the usual scope of LLM pharmacy research — narrow and long rather than wide and brief — and argues that longer, single-medication interactions expose a different class of errors than short multi-medication ones. It also argues that this kind of system "should be evaluated outside of commonly used benchmarks for LLMs, as these do not adequately capture the complexities of this kind of conversational system," and proposes HIE as an alternative.

• Real-world applications:

  • Pharmacy support for emergency contraceptive pills, a medication that is available without prescription but still requires counseling under Finnish law.
  • Social robots that support and promote medication safety at pharmacies, which is the broader project this work is grounded in.
  • A potentially less uncomfortable interaction for customers, since the paper notes that purchasing ECPs can be stressful and an agent interaction may be preferable to speaking with a human pharmacist.
  • A modular template: the authors state that ideally the conversation specification and knowledge base would be interchangeable with those of another medication, allowing the same system to counsel for different medicines.

• Industry relevance: The paper positions reliability and quality as design priorities for LLM systems generally, and identifies a design tension — flexibility versus adherence to conversational requirements — that it claims is relevant to any LLM-based system operating in a domain where errors are costly. It also notes that deployment without supervision involves considerable risk and recommends adaptive monitoring that starts with a human in the loop and scales back as the system proves itself.

Future Directions

• Finding design patterns for organizing multiple agents. The conclusion states that future work should especially focus on design patterns that organize multiple agents in ways that make them more reliable.
• Resolving task overlap between agents. The paper observes that there is currently some overlap in the tasks delegated between agents and that introducing additional agents might mitigate difficulties, but this would require further examination of how best to organize multiple agents in such a system.
• Developing and applying HIE evaluation. Evaluation is described as sorely needed but difficult, because it matters not only whether the customer received the correct recommendation but also how the system reached that conclusion and how it interacted with the customer. The HIE design framework cited consists of three stages: identifying risk or harm area, characterizing the use context, and choosing the evaluation parameters.
• Working out transparency, accountability and liability. It is not obvious how transparent the system must be, since a too-transparent system becomes too technical for a lay person while an opaque one builds mistrust and anxiety. The paper suggests a summarization transcript of the conversation — which could let domain experts validate the decision at the checkout desk — but states it is not clear exactly what information should be listed there, and that accountability, explainability and liability remain actively debated and unresolved.

Target Audience

Researchers and practitioners working on LLM-based conversational agents in healthcare and other high-stakes settings; human-computer interaction researchers studying multi-agent design, tool use and evaluation of multi-turn systems; pharmacists, pharmacy technology developers and health informatics teams; and anyone designing LLM systems that must satisfy fixed procedural or regulatory requirements while still handling unpredictable human language.

Authors’ abstract

The introduction of large language models (LLMs) has greatly enhanced the capabilities of software agents. Instead of relying on rule-based interactions, agents can now interact in flexible ways akin to humans. However, this flexibility quickly becomes a problem in fields where errors can be disastrous, such as in a pharmacy context, but the opposite also holds true; a system that is too inflexible will also lead to errors, as it can become too rigid to handle situations that are not accounted for. Work using LLMs in a pharmacy context have adopted a wide scope, accounting for many different medications in brief interactions -- our strategy is the opposite: focus on a more narrow and long task. This not only enables a greater understanding of the task at hand, but also provides insight into what challenges are present in an interaction of longer nature. The main challenge, however, remains the same for a narrow and wide system: it needs to strike a balance between adherence to conversational requirements and flexibility. In an effort to strike such a balance, we present a prototype system meant to provide medication counseling while juggling these two extremes. We also cover our design in constructing such a system, with a focus on methods aiming to fulfill conversation requirements, reduce hallucinations and promote high-quality responses. The methods used have the potential to increase the determinism of the system, while simultaneously not removing the dynamic conversational abilities granted by the usage of LLMs. However, a great deal of work remains ahead, and the development of this kind of system needs to involve continuous testing and a human-in-the-loop. It should also be evaluated outside of commonly used benchmarks for LLMs, as these do not adequately capture the complexities of this kind of conversational system.