A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agents

AI summary

Overview

Research area: Mechanistic interpretability and evaluation of large language model (LLM) agents, specifically goal-directedness in an agentic navigation setting (ICML keywords listed: goal-directedness, interpretability, evaluation, representation analysis, LLM agents, Machine Learning).

Technical level: Advanced. The paper combines behavioural benchmarking against optimal policies with activation probing and one-shot plan decoding, and assumes familiarity with Markov decision processes, A* search, and residual-stream interpretability methods.

Scope (1 sentence): The paper proposes and applies a white-box framework that pairs behavioural comparison to an optimal policy with probing of internal activations to assess whether GPT-OSS-20B, acting as an agent in a fully observable MiniGrid navigation task, represents and pursues its goal.

What This Paper Is About

The paper addresses the problem that there is no established methodology for reliably attributing goals to agentic systems: purely behavioural measures can be confounded by capability limits, and a system with misaligned internal goals could still produce aligned behaviour. The authors build a framework that evaluates goal-directedness by combining how an agent behaves against an optimal policy with what its internal representations encode, using an LLM agent navigating a 2D grid world to a goal state as the case study.

Key Contributions

1. A white-box framework that combines behavioural assessment with representation probing analyses for evaluating goal-directedness.
2. Controlled environment perturbations (iso-difficulty transformations) and multi-goal task structures designed to measure bias and robustness in the agent's goal-directed behaviour.
3. Probing of the agent's activations for environment state beliefs ("cognitive maps") and multi-step action plans, and use of those decoded representations to assess behavioural coherence relative to internal task-relevant information.
4. Release of code at github.com/SPAR-Telos/interp and github.com/SPAR-Telos/reveng, plus an interactive viewer for decoded cognitive maps at huggingface.co/spaces/project-telos/trace-viewer.

Main Findings

• Behaviour degrades with task difficulty: Per-action accuracy decreases monotonically with both grid size and obstacle density, while policy entropy and Jensen–Shannon divergence from the optimal policy increase with both. The authors conclude the policy becomes less optimal and more uncertain on harder grids.
• Distance-to-goal effect: Per-action accuracy decreases linearly with distance to the goal for distances below 20 steps, after which estimates become noisier; JSD from the optimal policy increases correspondingly, and variance in both metrics grows with distance. When controlling for other factors, accuracy decreases most systematically with grid size and obstacle density, with distance to goal playing a significant role only for larger grid sizes.
• Robustness to iso-difficulty transformations: Across four transformations (ReflectEnv, RotateEnv, StartGoalSwap, TransposeEnv), each preserving grid size, obstacle density, and optimal path length, no statistically significant differences appeared in any evaluated metric under a Wilcoxon signed-rank test. The authors read this as behaviour driven by task-relevant information rather than incidental grid configuration.
• Instrumental goals handled well: In KeyDoorEnv, the agent achieved a 100% success rate and 98.7% (± 3.2) accuracy, with stage-specific accuracy of 98.6% (± 5.7) for collecting the key, 99.2% (± 3.2) for opening the door, and 99.2% (± 3.3) for reaching the goal, and a 100% key pickup rate.
• Goal-like artefacts act as distractors: In KeyNoDoorEnv, where the key has no function, success was 98.9% and accuracy 97.2% (± 11.1), but the agent still picked up the key in 17% of trajectories, and 75% of non-optimal actions moved towards the key. In 2PathKeyEnv the agent was biased towards the key-containing path, with a 67.3% key pickup rate; that bias produced slightly higher per-action accuracy (76.0% ± 16.1 vs. 74.3% ± 15.7) but lower success than the no-key control (71.4% vs. 75.5%), and trajectories with and without the key showed low Jaccard similarity (65.6% ± 35.8).
• Environment is encoded non-linearly and coarsely: An MLP probe decoded cell identities at around 70% accuracy, reaching a maximum of 75.7% for 11×11 grids, while linear probes reached 39.1% in the same setting, suggesting non-linear encoding. Recall was especially high for goal (83–99%) and agent (72–100%) positions, but the probes assigned agent and goal labels to multiple cells near the true locations (high recall, low precision), and wall positions were not represented in detail.
• Agent and goal positions are recoverable approximately: Binary localisation accuracy drops as grid size increases, but the average Manhattan distance between predicted and true locations remained lower than 2 even for large 15×15 grids.
• Environment information degrades after reasoning: Overall probe accuracy for cognitive maps dropped from 75% before reasoning to 60% after reasoning, with notable decreases in agent, goal, and open recall and in agent and goal precision. The authors interpret this as representation reorganisation shifting from spatial cues towards action selection.
• Actions are broadly consistent with decoded beliefs: Accuracy against the optimal policy on the decoded cognitive map (Acc. Dec.) averaged 82.5% across grid sizes, and agreement between the decoded-grid and ground-truth-grid policies averaged 83.9%, always above 77% except for the highest density grids (57.6% at d = 1.0). Acc. Dec. was consistently lower than Acc. GT, which the authors attribute to the fuzzy agent/goal representations and the top-1 argmax collapsing uncertainty.
• Recovery of apparent failures: The recovery metric—actions suboptimal in the true environment but optimal under the decoded cognitive map—ranged between 37.4% and 88.4% (average 57.9%), highest in medium-to-large and lower-density grids, and lowest (37.4%) at d = 1.0.
• Uncertainty-aware decoding explains more actions: Against top-k decoded grids for k ∈ {3, 5, 10}, accuracy was consistently higher than against the ground truth grid across grid sizes and complexity bins, which the authors read as evidence the agent plans under a distribution over plausible states rather than a single state.
• Plans are partially decodable, non-monotonically by capacity: All plan probes exceeded the 0.25^N random baseline at short horizons. The 2-layer probe performed best overall, especially with post-reasoning activations, at 66.49% accuracy for N = 1 and strongest through N = 4. For longer prefixes the 2-layer pre-reasoning probe was strongest, at 7.3% (N = 5), 5.0% (N = 6), and 3.8% (N = 7). The 1-layer probe retained strong one-step decodability (45.5% pre-reasoning, 50.6% post-reasoning) but deteriorated rapidly with prefix length, and the 4-layer probe did not consistently beat smaller probes. The authors present this non-monotonicity as a control against the decoder itself solving the navigation task.
• Interventions require distributed editing: Preliminary activation patching on the residual stream changed the model's action distribution only when applied across all layers simultaneously, and only at grid state tokens or the token immediately preceding the action output; single-layer interventions on chat template tokens were ineffective.
• Metrics reported without added insight: The paper states that success rate and expected calibration error, while computed, did not provide additional insight beyond the metrics discussed.

Methodology in Plain English

The researchers selected GPT-OSS-20B for its manageable size and strong performance on complex tasks, and placed it in a fully observable MiniGrid environment as an agent that must reach a goal square one action at a time. The grid is converted into a text representation in which each cell maps to exactly one token, avoiding tokenisation artefacts, and the agent has full observability to remove memory, belief updating under perceptual uncertainty, and exploration–exploitation trade-offs as confounders (the authors note in the appendix that LLM agents, including frontier models, perform poorly in partially observable grid worlds with behaviours like redundant backtracking).

Behaviourally, they built grids at sizes {7, 9, 11, 13, 15} and obstacle densities {0.0, 0.2, 0.4, 0.6, 0.8, 1.0}, generating 10 random grids per size–density pair and evaluating 10 trajectories per grid at sampling temperature 0.7 with a maximum horizon of 1.5 × L, where L is the optimal path length. The reference policy comes from A* search using Manhattan distance. They measured per-action accuracy (fraction of actions in the optimal action set), entropy of action distributions, and Jensen–Shannon divergence from the optimal policy, estimating the agent's policy from the relative frequency of actions rather than action token log-probability, since the latter converges to 1 after the reasoning chain.

They then perturbed environments in ways that preserve difficulty (reflection, 90-degree rotation, swapping start and goal, transposition) and compared paired metrics with a Wilcoxon signed-rank test, and separately built multi-goal variants (KeyDoorEnv, KeyNoDoorEnv, 2PathKeyEnv) with 100 trajectories for the first two and 100 trajectory pairs for the third, using a horizon of 30.

For the representational side, they extracted residual-stream activations from the final three pre- and post-reasoning tokens in the model chat template (<|end|>, <|start|>, and assistant) at layers 7, 15, and 23. Cognitive map probes took inputs of the form [activation, x, y] and predicted a cell class from {agent, goal, wall, open, padding}, with smaller grids padded to 15×15 and minority classes upsampled; the MLP probe used two layers, ReLU, and a hidden dimension of 1024, trained with AdamW and weight decay after normalisation. Plan decoding used a shared bottleneck (linear projection to 1024 dimensions plus LayerNorm) followed by a Transformer decoder with T learned query embeddings that cross-attend over the three token activations, with 1, 2, or 4 layers and 8 heads per layer. Crucially, the decoder predicts the whole plan at once rather than autoregressively, so any coherent multi-step structure must already exist in the base model's activations; performance is measured by prefix accuracy, the fraction of episodes where the first N predicted actions exactly match the target prefix, on 3,000 training trajectories, a 600-trajectory validation set, and a 300-trajectory test set with T = 10.

Why This Matters

The paper argues that introspective examination is required beyond behavioural evaluations to characterise how agents represent and pursue their objectives, and that interpreting apparent failures through the agent's own decoded beliefs changes whether those failures look like a lack of goal-directedness or an artifact of a fuzzy world model. This matters for research because behavioural measures alone can be confounded by capability limits, and because a substantial fraction of apparent failures (recovery averaging 57.9%) are attributable to inaccurate representations rather than absent goal pursuit. It also matters because goal-like but functionally useless cues measurably biased the agent's path selection, illustrating that prompt-specified goals are not always the dominant influence on behaviour.

Real-world applications:

• Safety monitoring of agentic systems, where internal readouts could complement external behaviour logging.
• Evaluation and benchmark design for LLM agents, where difficulty-preserving transformations provide a way to test whether performance reflects task structure or incidental configuration.
• Debugging deployed agents whose world models are imperfect, by checking whether actions are consistent with the agent's decoded beliefs rather than ground truth.
• Auditing for distractor-driven behaviour, such as being pulled towards salient but irrelevant objects.

Industry relevance: the framework speaks to organisations building and shipping LLM agents that must act on goals reliably, and to those who need evidence about what an agent internally represents before deployment. The released code and the interactive cognitive-map viewer lower the barrier to reproducing and extending the analysis.

Future Directions

• Identifying layer- and position-specific intervention strategies: the authors state this is an important future direction, since readouts succeeded from individual layers while interventions required all-layer patching.
• Better handling of representational uncertainty: top-1 decoding collapses fuzzy agent/goal beliefs, and the authors suggest actions are better characterised as planning under a distribution over plausible states, motivating further work on uncertainty-aware decoding.
• Extending the framework beyond fully observable, single-agent grid worlds; the appendix discusses the difficulty of disentangling capability limits from failures of goal-directedness in partially observable settings.
• Clarifying why goal-like non-functional artefacts bias behaviour, which the authors conjecture reflects a conflict between the prompt-specified task goal and semantic associations from training (such as keys signalling progress in games).

Target Audience

Researchers and practitioners in LLM agent evaluation, AI safety, and mechanistic interpretability who want a concrete methodology linking behaviour to internal representations; also useful for engineers building or auditing agentic systems who need to reason about why an agent behaves suboptimally. Readers should be comfortable with reinforcement learning formalisms, probing classifiers, and transformer internals.

Authors’ abstract

Understanding an agent's goals helps explain and predict its behaviour, yet there is no established methodology for reliably attributing goals to agentic systems. We propose a framework for evaluating goal-directedness that integrates behavioural evaluation with interpretability-based analyses of models' internal representations. As a case study, we examine an LLM agent navigating a 2D grid world towards a goal state. Behaviourally, we evaluate the agent against optimal policies across varying grid sizes, obstacle densities, and goal structures, finding that performance scales with task difficulty while remaining robust to difficulty-preserving transformations and multi-goal structures. We then use probing methods to decode internal representations of the environment and multi-step action plans. We find that the LLM agent non-linearly encodes a coarse spatial map, preserving approximate task-relevant cues about its position and the goal location; that its actions are broadly consistent with these internal representations; and that reasoning reorganises them, shifting from spatial cues towards immediate action selection. Our findings support the view that introspective examination is required beyond behavioural evaluations to characterise how agents represent and pursue their objectives.