More Than Irrational: Modeling Belief-Biased Agents

AI summary

Overview

Research area: Artificial intelligence, specifically computational rationality (CR), user modeling, and human-AI collaboration.

Technical level: Intermediate. The paper builds on Partially Observable Markov Decision Processes (POMDPs), Bayesian filtering, and particle filtering, and assumes some familiarity with reinforcement learning concepts such as policies and action-value functions.

One-sentence scope: The paper formalizes users whose apparently irrational behavior arises from acting optimally under beliefs corrupted by bounded memory, and introduces an online inference method that recovers the user's latent cognitive bound and belief state from passive observation of their actions.

What This Paper Is About

AI assistants need to predict and infer the behavior of human users, but human behavior often looks sub-optimal. This paper argues that much of that behavior is not irrationality at all: it is rational decision-making performed under a biased belief state caused by a bounded memory process. The goal is to build a formal user model that explains such behavior and an inference method that identifies the user's latent memory bound online, from observed actions alone.

Key Contributions

1. A formal computational-rational (CR) user model for belief-biased agents. The authors introduce a class of models in which a parameterized internal memory process f_θ (with user-specific cognitive bound θ ∈ Θ) corrupts the agent's memory of past observations and actions. Because the memory itself can change over time (õ_t^j ≠ õ_{t-1}^j), the agent's belief is dynamically inconsistent, and computing it requires marginalizing over the entire corrupted memory sequence rather than a one-step Bayesian update.

2. An explicit demonstration that biased beliefs produce sub-optimal behavior under optimal acting. The CR agent acts optimally with respect to its own biased belief via a softmax policy π*(a|b̃;θ) = exp(τQ*(b̃,a;θ)) / Σ_a' exp(τQ*(b̃,a';θ)), yet to an external observer with the objective belief b*, the action can look sub-optimal (Q*(b*, a_CR) < Q*(b*, a_FR)).

3. An efficient online inference method based on nested particle filtering (NPF). The method jointly tracks the user's latent belief state h̃ and estimates the unknown cognitive bound θ from a stream of passively observed actions, using N_θ outer particles and N_h̃ conditional particles per bound. The authors report that exact inference costs O(|S|^t t!) while their method costs O(N_θ N_h̃ t |S|), with biased-belief computation costing O(t|S|).

4. A demonstration of downstream adaptive assistance via an Assistive-POMDP. Using the inference output as a belief over hidden user quantities, an AI assistant is trained with Proximal Policy Optimization (PPO) to choose between doing nothing, giving a memory hint at small cost, or giving a direct action hint at higher cost.

Main Findings

• The CR model generates a plausible spectrum of behaviors across memory capacities. In the T-maze task, with perfect memory (θ = 0.0) the agent acted optimally, taking one step down to observe the object and then proceeding directly to the correct terminal state. With θ = 0.4 the agent learned to collect more observations to memorize robustly. With θ = 0.7 it showed a "forget-recheck" pattern, returning to re-check the object after appearing to forget it. With θ = 1.0 (no memory) it did not explore and instead made a random turn.

• Online inference recovers the ground-truth bound accurately and efficiently. Averaged across all θ_true values and 5 random seeds with τ = 3.0, the final Posterior Mean (PM) error was 0.0087 ± 0.0035 (standard error). Relative to the PM error at t = 1, the PM error decreased by 90% at t = 45 and by 95% at t = 78, with most error reduction occurring in the first 20-30 steps (roughly 2-3 episodes).

• Identifiability is a real but surmountable challenge. For a user with θ_true = 0.4, the posterior initially concentrated around competing hypotheses {0.4, 0.6}, showing that different cognitive bounds can produce similar actions; with accumulating evidence the method converged to the true value (visualized at steps 1, 25, 50, 75, and 100).

• Inferred bounds drive an adaptive assistance policy. The learned assistant rarely intervened for users with θ ≤ 0.3, provided more memory hints and action hints for users with θ ∈ [0.4, 0.8], and provided more direct action hints for users with severe decay (θ ≥ 0.9), on the reasoning that such users would immediately forget simple memory hints.

• Assistance is timed to critical moments. For moderately forgetful users (θ ∈ {0.4, 0.6}), memory hints were provided occasionally around the end of episodes, right before the critical decision; for a user with no memory (θ = 1.0), only an action hint was provided on the last step.

• Stated limitations. The authors assume both the user model and the inference method have access to the underlying problem dynamics, and they note that their chosen memory model (memory decay) is likely insufficient to capture realistically sophisticated agents. Sensitivity analysis of N_h̃ and τ is stated to be available in the extended version in the code repository.

Methodology in Plain English

The authors begin from the POMDP framework, in which an ideal agent maintains an accurate belief about hidden states through Bayesian filtering and acts to maximize expected return. They then modify this picture: instead of remembering the true history of observations and actions, the user keeps an internal, corrupted memory that evolves according to a stochastic memory process f_θ, controlled by a bound parameter θ. Because the memory can be rewritten retroactively, the user's belief no longer satisfies the Markov property, and the belief must be recomputed by marginalizing over the whole corrupted memory sequence.

The user still acts optimally — but optimally with respect to the corrupted belief, not the true world. This mismatch is what makes the behavior look irrational to an outside observer.

The inference problem is to recover θ and track h̃ from observed action-observation histories. Because the exact posterior sum over memory sequences is intractable, the authors use nested particle filtering: a set of outer particles represents candidate values of θ, and for each of those, an inner set of particles represents possible corrupted memory states. Each particle is scored by how likely the user's last action would be under that particle's biased belief, and the weights are updated accordingly.

Validation is done in a grid-world T-maze navigation task: the agent starts in a hallway, must explore a bottom room to observe an object, then navigate to the matching terminal arm of the "T". The hidden state includes the agent's position and the target object; actions are four directions plus staying in place; observations are a 3×3 grid around the agent; transitions are deterministic; and reaching the correct terminal state yields a reward of 1 − 0.9 × timesteps/maxsteps, with 0 otherwise. Memory decay is implemented as a p = θ chance that any observation of the target object in the buffer is replaced by a default value. Optimal policies for a range of θ values were pre-trained with PPO. Inference was tested on 100-step trajectories with θ_true sampled uniformly from {0.0, 0.1, …, 1.0} and a uniform prior, evaluated by PM error and MAP error.

Why This Matters

Impact on research. The paper reframes a large class of "irrational" behavior as rational action under corrupted beliefs, and treats bounded memory — which the authors argue has received little attention in general form within computational rationality — as a first-class latent variable. It also provides a tractable inference formulation and an explicit complexity comparison against exact inference.

Real-world applications:

• Adaptive AI assistants. An assistant can decide whether a user needs a subtle reminder or direct action guidance based on an inferred memory bound, rather than assuming the user is simply irrational.
• Assistive technologies for memory-impaired users. The T-maze assistive demonstration models the trade-off between low-cost memory hints and higher-cost direct action suggestions.
• Human-AI collaboration in sequential decision tasks. Wherever an AI must anticipate a collaborator's future actions from past behavior — the paper motivates human-AI collaboration directly.
• User modeling where sub-optimal inputs are common. The framework explains, for example, why someone searching for a misplaced phone might check places that seem irrelevant because their belief about the location is biased.

Industry relevance. Any system that must predict user actions in real time benefits from a user model that infers latent constraints rather than discarding seemingly noisy behavior. The authors emphasize that their O(N_θ N_h̃ t |S|) method "allows for real-time inference," and they report that most of the error reduction happens within roughly 2-3 episodes, which matters for deployment settings with limited observations.

Future Directions

• Removing the assumption of known dynamics. The authors state that they assumed both the user model and the inference method have access to the underlying dynamics of the problem; relaxing this is flagged as future work.
• Richer memory models. The authors note that their choice of memory model is reasonable but likely insufficient to capture realistically sophisticated agents, motivating more expressive f_θ formulations.
• Broader general modeling of memory-driven behavior. The paper positions itself as a step toward a general model for prediction and inference over irrational behavior caused by imperfect memory, rather than an application-specific solution.
• Additional latent variable forms. The discussion highlights that the model is flexible in its choice of memory model f_θ and allows flexibility in the form of latent variables, leaving the exploration of alternative latent variables open.

Target Audience

Researchers and practitioners working on computational rationality, cognitive modeling, user modeling, and human-AI collaboration will benefit most. It is also relevant to reinforcement learning researchers interested in POMDPs and sequential Monte Carlo inference, and to engineers building adaptive assistants or assistive technologies that must reason about users' cognitive limitations. Readers without a background in POMDPs or particle filtering will need to consult the cited foundational work, since the paper presents the formalism directly.

Authors’ abstract

Despite the explosive growth of AI and the technologies built upon it, predicting and inferring the sub-optimal behavior of users or human collaborators remains a critical challenge. In many cases, such behaviors are not a result of irrationality, but rather a rational decision made given inherent cognitive bounds and biased beliefs about the world. In this paper, we formally introduce a class of computational-rational (CR) user models for cognitively-bounded agents acting optimally under biased beliefs. The key novelty lies in explicitly modeling how a bounded memory process leads to a dynamically inconsistent and biased belief state and, consequently, sub-optimal sequential decision-making. We address the challenge of identifying the latent user-specific bound and inferring biased belief states from passive observations on the fly. We argue that for our formalized CR model family with an explicit and parameterized cognitive process, this challenge is tractable. To support our claim, we propose an efficient online inference method based on nested particle filtering that simultaneously tracks the user's latent belief state and estimates the unknown cognitive bound from a stream of observed actions. We validate our approach in a representative navigation task using memory decay as an example of a cognitive bound. With simulations, we show that (1) our CR model generates intuitively plausible behaviors corresponding to different levels of memory capacity, and (2) our inference method accurately and efficiently recovers the ground-truth cognitive bounds from limited observations ($\le 100$ steps). We further demonstrate how this approach provides a principled foundation for developing adaptive AI assistants, enabling adaptive assistance that accounts for the user's memory limitations.