Exploration Through Introspection: A Self-Aware Reward Model

AI summary

Overview

Research area: Reinforcement learning, Theory of Mind in AI, intrinsic motivation and exploration, computational models of affect and pain perception.

Technical level: Intermediate. The ideas are conceptual and biologically motivated, but the implementation leans on standard tools (hidden Markov models, Q-learning, gridworld benchmarks) that a reader familiar with basic RL will recognise.

Scope: The paper introduces a "self-aware" reward model in which a Q-learning agent infers a latent belief about its own pain from its own happiness signal, and uses that belief as an intrinsic exploration incentive in stationary and non-stationary gridworlds.

What This Paper Is About

Most AI work on Theory of Mind focuses on inferring the mental states of others. This paper asks whether the same kind of inference applied to oneself — the agent inferring its own internal aversive state — can serve as a useful learning signal. The authors build an agent that treats "pain" not as feedback from the environment but as a hidden state it must infer from its own happiness, then measure whether this self-awareness improves how quickly and how often the agent finds food in gridworlds.

Key Contributions

1. An introspective exploration component. A hidden Markov model (HMM) runs online inside the agent, using the forward algorithm to infer a "pain-belief" — the probability of being in the hidden state pain — from observations derived from the agent's own happiness value.
2. A subjective reward function called well-being. The authors extend the happiness function of Dubey et al. (2022), which combines objective reward, expectation, and comparison, by subtracting a weighted pain term, giving f^w = f^h − w4 · Pain.
3. Two perception models compared. Using HMM parameters adapted from Eckert et al. (2022), the authors instantiate a "normal pain" model (transitions favour recovery, emissions distinguish noxious from harmless) and a "chronic pain" model (sticky transitions, ambiguous emissions), and compare agents using each against a "No pain" baseline.
4. A computational account of relief-seeking behaviour. The chronic pain agent accumulates negative well-being across its lifetime and only reaches approximately zero well-being when it is on the food state, which the authors interpret as a parallel to negative reinforcement in addiction.

Main Findings

• Introspection generally helps. Introspective agents ("Normal pain" and "Chronic pain") generally outperform "No pain" baseline agents within the same reward category, in both the stationary and non-stationary environments, with statistically significant improvements marked in Figure 1, Figure 5, Table 1 and Table 2 (one-sided paired-samples t-test, p ≪ 0.05).
• Normal and chronic pain tie in the stationary environment's 'Objective+Expect' category. Normal pain reached M = 2295.6, SD = 65.7; chronic pain reached M = 2295.0, SD = 66.1. The "No pain" baseline in that category scored M = 1973.1, SD = 385.0.
• Chronic pain wins in the non-stationary 'Objective+Expect' category. Chronic pain scored M = 4214.6, SD = 165.4 versus normal pain at M = 3814.0, SD = 446.6, against a "No pain" baseline of M = 2371.0, SD = 613.3.
• Opposite optimal learning rates in the non-stationary 'Objective only' category. The normal agent performed best at α = 0.9 (M = 3101.8, SD = 271.8) and the chronic agent at α = 0.1 (M = 4142.5, SD = 177.2), compared with the "No pain" baseline at M = 1586.5, SD = 631.2 with α = 0.9.
• Success despite negative well-being. The chronic pain agent outperformed its counterpart while accumulating a negative cumulative well-being across its lifetime; its momentary well-being only returns to approximately zero when it reaches the food state.
• Chronic pain is also the weakest in one setting. Within the non-stationary 'Objective+Expect' category, agents with this maladaptive component also yielded the worst overall performance (Appendix C, Figure 6), even though its best member beat the normal pain best member.
• Normal pain acts like a low-pass filter. The authors describe the normal model as smoothing the happiness signal into a stable belief state that supplies a dynamic exploration bonus, which is particularly useful in non-stationary environments.

Methodology in Plain English

The agents live in a 7×7 gridworld containing one food state worth +1 objective reward (all other states give 0), and they spawn in the bottom left corner with five possible actions per step: UP, DOWN, LEFT, RIGHT and STAY. In the stationary environment the food never moves and the agent's lifetime is 2500 steps; in the non-stationary environment the food moves to another corner every 1250 time steps during a 5000-step lifetime.

Each agent learns with Q-learning and an ε-greedy policy. Its reward is the "well-being" function f^w, a weighted combination of an objective reward term, an expectation term, and a comparison term, minus a weighted pain term. The pain term is not observed directly: at each step the agent converts its happiness value into a binary observation (harmless if happiness is at least 0, noxious if below 0) and feeds it to a hidden Markov model whose hidden states are pain and no_pain. The forward algorithm gives the online belief that the agent is in pain, and that belief is what enters the reward.

Two HMM parameter sets are used. The normal model has transition probabilities of 0.3/0.7 from pain and 0.2/0.8 from no_pain, emissions of 0.8/0.2 for pain and 0.1/0.9 for no_pain, and initial state 0.223 pain / 0.777 no_pain. The chronic model has transitions of 0.8/0.2 and 0.7/0.3, identical emissions of 0.6/0.4 for both hidden states, and initial state 0.777 pain / 0.223 no_pain.

To find the best agents, the authors use the optimal reward framework of Singh et al. (2009) with mean cumulative objective reward (COR) as the fitness metric. Reward functions are grouped into seven categories ('Objective only', 'Expect only', 'Compare only', 'Objective+Expect', 'Objective+Compare', 'Expect+Compare', 'All') and three pain subcategories (No pain, Normal pain, Chronic pain). Search ranges were w1–w4 ∈ [0, 0.1, 0.3, 0.5, 0.7, 0.9, 1], ρ ∈ [0.01, 0.05, 0.1, 0.3, 0.5, 0.7, 0.9, 1], α ∈ [0.1, 0.3, 0.5, 0.7, 0.9] and ε ∈ [0.01, 0.1], giving 312,130 subjective reward functions per environment after removing duplicates and invalid functions. Every configuration was evaluated over n = 300 histories with a discount factor of γ = 0.99.

Why This Matters

Impact on research. The paper connects Bayesian models of pain perception, subjective/expectation-based reward functions, and Theory of Mind in a single RL agent. It offers a concrete way to test the hypothesis that humans use a unified system for thinking about mental states, whether directed at the self or at others. It also introduces chronic versus normal pain perception as a controllable variable that produces measurably different learning dynamics and psychologically interpretable behaviour.

Real-world applications (as suggested by the framing, not experimentally demonstrated in the paper):

• Intrinsic motivation for agents operating in non-stationary environments where reward goals shift over time.
• Affect-based reward shaping for interactive agents that need to adapt quickly after a change in their surroundings.
• Computational psychiatry models, where maladaptive perception models generate testable behavioural signatures such as relief-seeking.
• Simulation of maladaptive consumption cycles (for example, addiction-like dynamics) as a tool for studying negative reinforcement.

Industry relevance. For teams building agents that must keep exploring after their environment changes, the paper suggests that modelling an internal aversive state can act as a cheap, self-generated exploration bonus that does not depend on hand-crafted novelty measures. The chronic pain result is a caution: a maladaptive perception model can boost task metrics while the agent's own well-being measure goes persistently negative, which matters for anyone using subjective reward components as proxies for agent welfare.

Future Directions

• Extend introspection to other minds. The authors state that their framework models only the self-application aspect of the hypothesised unified system, and that future work can test the unified account by extending the architecture to infer others' states.
• Move beyond the simple ε-greedy baseline. The authors list the ε-greedy baseline as a limitation of the current work.
• Relax the fixed HMM parameters. Fixed HMM parameters are also named as a limitation, leaving open what happens when the perceptual model itself is learned.
• Investigate why chronic pain helps adaptation. The paper shows that the chronic model adapts faster and achieves higher cumulative objective reward in the non-stationary environment, but also produces the worst performance within the 'Objective+Expect' category and persistently negative well-being; the conditions under which a maladaptive perception model is beneficial are not fully resolved.

Target Audience

Researchers and graduate students working on reinforcement learning, intrinsic motivation and exploration, Theory of Mind in AI, or computational models of affect and pain. It is also relevant to readers in computational psychiatry or cognitive science who are interested in translating models of maladaptive perception into agent architectures, and to practitioners who want a self-generated exploration signal for non-stationary environments.

Authors’ abstract

Understanding how artificial agents model internal mental states is central to advancing Theory of Mind in AI. Evidence points to a unified system for self- and other-awareness. We explore this self-awareness by having reinforcement learning agents infer their own internal states in gridworld environments. Specifically, we introduce an introspective exploration component that is inspired by biological pain as a learning signal by utilizing a hidden Markov model to infer "pain-belief" from online observations. This signal is integrated into a subjective reward function to study how self-awareness affects the agent's learning abilities. Further, we use this computational framework to investigate the difference in performance between normal and chronic pain perception models. Results show that introspective agents in general significantly outperform standard baseline agents and can replicate complex human-like behaviors.