Modulation of temporal decision-making in a deep reinforcement learning agent under the dual-task paradigm

AI summary

Overview

Research area: Cognitive-inspired AI / deep reinforcement learning, specifically the study of temporal decision-making and timing interference under a dual-task paradigm.

Technical level: Intermediate. Readers should be comfortable with reinforcement learning basics (agents, rewards, episodes), recurrent networks (LSTM), and standard analysis tools such as PCA and FFT. The cognitive-science framing (time production, dual-task interference) is explained from first principles.

Scope: A single study training separate deep reinforcement learning agents on a single task and a dual task built from a simplified Overcooked environment, then comparing their timing behavior and inspecting their LSTM activations for evidence of a time-keeping mechanism.

What This Paper Is About

Humans reliably misjudge time when they have to do something else at the same time: adding a concurrent cognitive task to a timing task typically makes people overproduce durations. This paper asks whether the same interference effect arises in an artificial agent. The authors embed a time production task inside a simplified Overcooked grid world, train one deep reinforcement learning agent on that task alone and another on the same task plus a concurrent number comparison task, and then compare when each agent checks its oven and how their recurrent network activity behaves.

Key Contributions

1. A dual-task adaptation of a simplified Overcooked environment, where an oven with an invisible internal timer enforces a time production task and an optional number comparison counter provides a concurrent cognitive load.
2. A demonstration that a DRL agent trained on the dual task (T+N) significantly overproduces the target durations relative to its single task (T) counterpart, across four target durations (7, 8, 9, and 10 time steps), with p < 0.001.
3. The claim that this is the first study to suggest DRL agents exhibit temporal interference effects in a dual-task paradigm similar to those observed in human timing research, and that the behavior was emergent rather than built in.
4. A preliminary neural analysis of the agents' LSTM layers using PCA and FFT, which found no conclusive evidence of a dedicated or intrinsic timer.

Main Findings

• Dual-task overproduction: The dual task (T+N) agents produced significantly longer "first oven checks" than the single task (T) agents, with p < 0.001, and this result was consistent across all four target durations (7, 8, 9, and 10 time steps).
• Comparable task performance: Despite being trained for the same number of training steps, the dual task agents achieved performance comparable to their single-task counterparts. In the least favorable case (target duration 10), the dual task (T+N) agent produced roughly 53% of the soups produced by its single task (T) counterpart.
• No time pressure confound in the design: The authors note that in the dual tasks the agents had enough time after the number game stopped to reach the oven and interact with it at least one time step before the target duration, and that the number comparison counter ran for 4 time steps irrespective of target duration.
• No evidence of a dedicated timer: PCA of the LSTM hidden state activations across the first 100 time steps showed complex oscillatory dynamics with multiple peaks within each cycle, reset at the onset of each new trial (after soup delivery), indicating information about trial boundaries but no clear timer.
• Frequency patterns diverge by task: In the FFT of the first principal components, for target durations 7 and 10 the single task (T) agents showed oscillations with frequencies equal to or greater than the target duration (suggesting accurate timing or underproduction), while the dual task (T+N) counterparts showed oscillations with frequencies lower than the target duration (suggesting overproduction). The authors state no clear link can be established between these peaks and an intrinsic time-keeping mechanism.
• The LSTM appears necessary for time-keeping: In a separate variant of the DRL agents not described in this study, removing the LSTM layer left the agent able to perform the single task (T) successfully, but it showed no time-keeping behavior, instead repeatedly performing the "Interact" with oven action and relying solely on state changes.
• Counter-like activity absent: Previous work with a DRL agent performing a dedicated time reproduction task had shown counter-like neural activity in the PCA of the LSTM layer; the authors attribute its absence here to the timing task being embedded in the soup delivery task with no immediate reward for correctly producing the target interval.

Methodology in Plain English

The researchers built a small grid world version of Overcooked. The agent moves around a 5x3 grid containing an onion dispenser, an oven, and a delivery counter, using six actions: wait, up, down, left, right, and interact. To score, it picks up an onion, puts it in the oven, must wait until the soup is cooked, then takes the soup out and delivers it. Each delivery gives a reward of +1; every other action gives zero reward, even if the soup sits in the oven longer than needed. Each episode lasts 100 time steps and the goal is to deliver as many soups as possible, so timing matters.

The oven starts an invisible timer when it receives an onion. The soup only becomes ready after a target duration, and interacting before that has no effect. The researchers trained agents for four target durations separately: 7, 8, 9, and 10 time steps.

In the dual task variation, a number comparison counter activates for 4 time steps after the onion goes in the oven. It shows a number between 1 and 10, and the agent must press "interact" if the number is less than 5 or "wait" if it is 5 or more. Correct answers give an immediate +1 reward; otherwise zero. After the four time steps the task continues exactly as in the single task setup.

All agents used the recurrent policy variant of Proximal Policy Optimization (PPO) from the Stable-Baselines3 library. The architecture was a CNN layer (mostly 1x1 convolutions) for the grid, an LSTM layer with 256 hidden units, and an MLP with 64 hidden units. An entropy coefficient of 0.05 encouraged exploration, which the authors say was especially helpful for training the dual-task variant. Each agent was trained for 100,000 time steps, and four independently trained agents were produced per task type.

To measure timing behavior, the authors defined the "first oven check": the oven timer value at which the agent first initiates the "Interact" action with an onion-carrying oven. If the timer equals or exceeds the target duration, the soup can be taken out and that value is recorded. If the timer is below the target, the value is recorded only when the agent keeps interacting consecutively until it can take the soup out; if any action in that run is not "Interact," nothing is recorded. The intuition is that the agent checks the oven because it believes the target duration has been reached. Distributions of first oven checks were compared across 25 episodes of 100 time steps each, using independent t-tests. For the neural analysis, the authors applied PCA to the LSTM hidden state activations over the first 100 time steps and then an FFT on the first principal components.

Why This Matters

Impact on research. The work sits at the intersection of cognitive science and deep learning, contributing to an effort to draw parallels between deep neural networks and biological systems. The authors note that such research is extensive in vision and audio but limited for temporal processing. If a DRL agent can reproduce a well-known human timing bias without being given any timer or biologically inspired structure, that suggests dual-task timing interference may be a general property of learning systems rather than something specific to biological brains. The negative neural result is also informative: it shows that reproducing a behavioral signature does not automatically come with an interpretable mechanism, and that finding one requires further work.

Real-world applications.

• Robotics and autonomous systems that must juggle time-sensitive actions with concurrent cognitive demands, such as a warehouse or kitchen robot waiting on a process while handling another task.
• Human-machine interaction design, where understanding when a system's internal timing drifts under load could inform safer or more predictable behavior.
• Cognitive modeling and computational psychiatry research, where agent-based models of timing interference could serve as testbeds for hypotheses about attention and time perception.
• Reinforcement learning benchmarking, since the task provides a controllable setting for probing whether emergent timing biases appear in learned policies.

Industry relevance. Any deployed RL or learned-control system that must act on timing while also handling secondary tasks can inherit systematic timing errors. Knowing that a concurrent task shifts timing behavior in a consistent direction, and that the network's recurrent state carries no obvious timer to inspect, matters for verification, interpretability, and safety arguments in industrial deployments.

Future Directions

• Improve the reward structure by using a delayed reward in the dual-task setting, which the authors say would better align with corresponding human timing studies and help verify the biological similarities in the observed behavior.
• Extend the study beyond the four target durations (7, 8, 9, and 10 time steps) to a wider range, which the authors state could further substantiate the findings.
• Broaden the neural analysis beyond the LSTM layer, since the current analysis focused exclusively on the LSTM layer and was preliminary, to include other layers of the DRL agent's architecture.
• Determine whether the oscillations observed in some target durations actually correspond to a time-keeping mechanism, since the authors report no conclusive neural evidence to explain the behavioral differences and state that further investigation is needed to understand the agents' underlying time-keeping mechanisms.

Target Audience

Researchers and graduate students working on cognitive-inspired AI, interpretability of deep reinforcement learning, or computational models of time perception. It is also relevant to cognitive scientists and psychologists interested in whether dual-task timing effects have analogs in artificial systems, and to RL practitioners who want a concrete example of emergent timing behavior and how to probe it. Readers without a background in reinforcement learning or cognitive timing research will find the paper accessible but should expect to consult the cited literature for the human timing findings it builds on.

Authors’ abstract

This study explores the interference in temporal processing within a dual-task paradigm from an artificial intelligence (AI) perspective. In this context, the dual-task setup is implemented as a simplified version of the Overcooked environment with two variations, single task (T) and dual task (T+N). Both variations involve an embedded time production task, but the dual task (T+N) additionally involves a concurrent number comparison task. Two deep reinforcement learning (DRL) agents were separately trained for each of these tasks. These agents exhibited emergent behavior consistent with human timing research. Specifically, the dual task (T+N) agent exhibited significant overproduction of time relative to its single task (T) counterpart. This result was consistent across four target durations. Preliminary analysis of neural dynamics in the agents' LSTM layers did not reveal any clear evidence of a dedicated or intrinsic timer. Hence, further investigation is needed to better understand the underlying time-keeping mechanisms of the agents and to provide insights into the observed behavioral patterns. This study is a small step towards exploring parallels between emergent DRL behavior and behavior observed in biological systems in order to facilitate a better understanding of both.