Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila

AI summary

Overview

Research area: NeuroAI / computational neuroscience, combining intrinsically motivated reinforcement learning, biomechanical simulation of Drosophila melanogaster, and whole-brain neural recordings.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (PPO, actor–critic, intrinsic rewards), unsupervised skill discovery, recurrent networks, and neural decoding methods.

Scope (one sentence): The paper tests whether hierarchical, intrinsically motivated artificial agents embodied in a neuromechanical digital twin of the fruit fly can quantitatively reproduce both the spontaneous turning behavior of real flies and the fly brain's neural prediction of future turns seconds in advance.

What This Paper Is About

Animals explore and act on their own, without external rewards or sensory cues, but the computational objective behind this autonomy is unknown. The authors ask whether specific "intrinsic motivation" algorithms — curiosity, entropy, and competency objectives — can generate behavior and internal representations that match real flies. They use a fly digital twin in a physics simulator plus a prior whole-brain recording study showing that a fly brain region signals future turn direction roughly 15 to 8 seconds before the turn, and they test which intrinsic objective best reproduces both the behavior and that neural precursor.

Key Contributions

1. A hierarchical, embodied testbed for intrinsic motivation. The authors implement intrinsically motivated agents in NeuroMechFly v2 (a neuromechanical digital twin of an adult female fly simulated in FlyGym with MuJoCo), where 20 intrinsic objectives operate over seven ethologically motivated behavioral primitives rather than raw joint torques.
2. Evidence that representations of future action emerge without any external task. Several intrinsically motivated agents develop internal states that predict turn direction seconds in advance, paralleling the prospective signal observed in the fly Inferior Posterior Slope (IPS), despite receiving no extrinsic reward and no fly data.
3. Identification of DIAYN as the best joint account of fly behavior and neural dynamics. Unsupervised skill discovery (DIAYN) hierarchically operating on behavioral primitives jointly reproduces future-action predictivity, the relationship between neural and behavioral predictability, inter-animal variability in turning behavior, and low-dimensional activity structure related to forward and angular velocity.
4. A demonstration that mean statistics and behavioral predictability alone are insufficient to distinguish models. Objectives that match average turning bias, or that produce trivially predictable behavior, can still fail to reproduce the fly's prospective internal signal; only joint evaluation of behavioral variability and internal future-action predictivity separates DIAYN from alternatives.

Main Findings

• Neural prospective coding: The biological fly IPS reaches neural prospective coding of 66.5 ± 1.4% balanced accuracy in the −15 to −8 s window. DIAYN reaches 82.1 ± 0.9% and NGU reaches 65.4 ± 1.8%, approaching or exceeding the fly.
• Behavioral prospective coding: Decoding from mean angular and forward velocity alone gives 60.5 ± 0.9% for flies, versus MAX at 92.2 ± 5.9%, DIAYN at 74.7 ± 1.4%, γ-progress at 72.4 ± 1.2%, and NGU at 66.3 ± 9.4%.
• Failure modes are common: Nine of the 20 intrinsic-drive methods produce no turns, or turns in only one direction, during evaluation — itself a behavioral mismatch with the fly, and a condition that precludes stratified cross-validation.
• The neural-to-behavioral ratio matters: DIAYN and NGU are the only trained methods that match or exceed both the neural and behavioral predictability of the fly, and only DIAYN preserves the fly's ratio between the two.
• Population-level behavioral diversity: DIAYN's skills span a much wider range of turning biases than other methods. APT and random controls have means close to the flies' but do not reproduce the fly population's spread. CSF and Random-DIAYN, which use the same one-skill-per-rollout schedule, show far narrower bias distributions, indicating the spread comes from the learned objective rather than the sampling scheme.
• Sum-difference encoding of kinematics: In trained DIAYN, the difference of a selected principal-component pair correlates with angular velocity (r = 0.153) and their sum with forward velocity (r = 0.413), comparable to the fly IPS (r = 0.191 and 0.429). In Random-DIAYN the best pair reaches only r = 0.021 and 0.113.
• Interpretation: The authors propose that individual behavioral variability may be a population-level strategy for maximizing state-space coverage, and that prospective activity reflects the persistence of an exploratory strategy rather than moment-by-moment movement.

Methodology in Plain English

The team built a virtual fly using an existing biomechanical model with articulated legs, compound-eye vision, smell, and proprioception. Rather than letting a network control every joint, they split control into two levels: every 100 ms a high-level policy chooses one of seven behaviors — forward, left, hard left, right, hard right, stop, or groom — and fixed low-level controllers (a central pattern generator driven by step trajectories recorded from real walking flies, plus a kinematic replay for grooming) translate that choice into joint torques.

They trained 20 different intrinsic-motivation policies, spanning curiosity-style methods (e.g., ICM, RND, NGU, Disagreement, MAX, γ-progress, C-TeC), entropy-style methods (APT, Proto-RL, RE3), and competency methods including empowerment variants and skill discovery (DIAYN, DADS, VALOR, VIC, CSF). All shared the same architecture (modality-specific encoders producing a 310-dimensional feature vector, actor and critic LSTMs of 256 units each), the same procedurally generated training arenas with three odor sources and 12 to 50 obstacles, 2,048 parallel environments, Recurrent PPO, and a budget of 50 million policy decisions. No external reward was given; randomly initialized policies and a policy trained on an extrinsic odor-navigation task served as controls.

Policies were then frozen and evaluated in a virtual version of the Brezovec et al. tethered-ball assay: a head-fixed fly on a 9 mm air-supported ball, with vision and odor zeroed to match the dark, odor-free biological experiment. Each policy produced nine 30-minute rollouts, matching the nine flies and 270 minutes of biological data. Velocities were smoothed with a 500 ms Savitzky–Golay filter, and turns were detected as peaks in absolute angular velocity above 200 °/s. Turning bias was summarized as log(N_R/N_L). A logistic regression decoder (elastic-net penalized, five-fold stratified cross-validation, balanced accuracy) was fit to a −15 to −8 s window to predict upcoming turn direction from network activations or from behavior, with hyperparameters tuned once on fly data and fixed for all agents. Finally, PCA was applied to each agent's best-decoding layer to test whether component sums and differences relate to forward and angular velocity the way the fly's two IPS hemispheres do.

Why This Matters

Impact on research. The paper extends the "goal-driven modeling" approach of computational neuroscience — normally applied to sensory systems — to intrinsic motivation and autonomous behavior. It shows that behavioral variability and predictive neural activity can arise from a single self-supervised objective, and it argues that models should be judged on neural and behavioral signatures jointly, not on downstream task performance alone. The framework provides a normative, non-imitation-based account of spontaneous behavior in an embodied agent.

Real-world applications (as motivated by the framework, not demonstrated in the paper):

• Designing autonomous robots or agents that explore and build useful representations without hand-specified reward functions.
• Using neural and behavioral signatures from animals as validation criteria for artificial agents, rather than task benchmarks alone.
• Building better virtual animal models for neuroscience experiments, where candidate objectives can be screened before wet-lab work.
• Informing population-level strategies in multi-agent systems, where diversity across agents may compensate for limited individual capacity.

Industry relevance. The work speaks to practitioners of reinforcement learning who struggle with reward specification, and to groups building embodied AI or digital twins of organisms for simulation-based testing. The finding that matching average behavior statistics is insufficient — and that internal-representation checks add discriminative power — is directly relevant to evaluating whether a model has captured the right mechanism rather than merely the right output distribution.

Future Directions

• Learn the primitives rather than fixing them. The authors note that constraining behavior to predefined locomotor primitives and low-level controllers limits agents largely to 2-D locomotion; jointly learning the primitives with their high-level selection would let the intrinsic objective shape its own low-level repertoire.
• Expand the behavioral repertoire. Adding flight is suggested as an extension, potentially building on the existing implementation of Vaxenburg et al. (2025).
• Move from IPS to whole-brain dynamics. The current simple training environment sufficed for the IPS comparison, but accounting for whole-brain locomotion dynamics would require an ethologically relevant environment with variable terrain, manipulation opportunities, and resources, which may favor different intrinsic objectives.
• Test the population-coverage hypothesis. The idea that individual behavioral handedness exists because diverse strategies collectively maximize state-space coverage at the population level is offered as a hypothesis, not a tested claim, leaving room for direct experimental and computational tests.

Target Audience

This paper is most valuable to researchers at the intersection of computational neuroscience and reinforcement learning — particularly those working on NeuroAI, intrinsic motivation, unsupervised skill discovery, and embodied agents. It also suits neuroscientists studying spontaneous behavior and motor planning in Drosophila, and engineers building autonomous agents who want biologically grounded alternatives to extrinsic reward design. Readers without a background in reinforcement learning or neural decoding will find the methods section demanding, though the framing and conclusions are accessible.

Authors’ abstract

Animals have a remarkable capacity for autonomy---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. We combine whole-brain recordings in the fruit fly (Drosophila melanogaster), which reveal neural representations of spontaneous exploratory decisions more than 10 seconds in advance, with a neuromechanical digital twin to test whether optimizing intrinsic goals yields realistically embodied artificial agents that quantitatively match fly behavior and internal representations of future volitional action during exploration. Our hierarchical policy selects from a discrete library of low-level motor primitives, consistent with ethological theories of behavioral syllables. We train agents without external reward using 20 intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. The agents are evaluated in a virtual tethered-ball assay modeled on the fly experiment, comparing spontaneous locomotor statistics and internal representations and decoding future turn direction from kinematics and neural activity on the same timescales observed in flies. DIAYN, a hierarchical competency algorithm learning discrete policies that maximize behavioral entropy and the separability of induced sensory observations, achieves the strongest joint behavioral and neural prospective coding performance relative to flies. Its learned policies closely capture individual variability in fly exploration strategies, suggesting that this diversity serves to maximize state-space coverage at the population level. DIAYN's latent activity also closely captures the sum-difference rule for interhemispheric encoding of kinematics in biological flies. These results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.