Comparing Dynamical Models Through Diffeomorphic Vector Field Alignment

AI summary

Overview

Research area: Theoretical neuroscience and scientific machine learning — specifically the analysis and comparison of learned dynamical systems models (recurrent neural networks used as models of brain dynamics).

Technical level: Advanced. The paper builds on dynamical systems theory (smooth orbital equivalence, diffeomorphisms, pushforward vector fields, limit sets) and on deep learning tools (Neural Ordinary Differential Equations, auto-differentiation).

Scope in one sentence: The paper introduces DFORM, a method that learns a nonlinear coordinate transformation (diffeomorphism) between the state spaces of two dynamical systems so their vector fields can be aligned and compared, and shows this framework can also locate low-dimensional dynamical motifs inside high-dimensional models.

What This Paper Is About

Dynamical models such as recurrent neural networks (RNNs) are widely used in theoretical neuroscience, but comparing the dynamics they learn is hard because their coordinate systems are not shared a priori, and because important features like saddle limit sets cannot be located analytically in high-dimensional nonlinear systems. The authors propose DFORM (Diffeomorphic vector field alignment FOR learned Models), which learns a nonlinear coordinate transformation that maps one system's trajectories onto another's in a maximally one-to-one fashion. The goal is to determine whether two models are mechanistically similar despite differing coordinate systems, and to locate mechanistically important low-dimensional structures embedded in high-dimensional models.

Key Contributions

1. An operationalized framework for smooth orbital equivalence. The authors derive a first-principles loss function, the orbital similarity loss, using the concept of a pushforward vector field, making the classical notion of smooth orbital equivalence usable in practice for learned models.

2. A Neural ODE-based construction of the diffeomorphism. The diffeomorphism is modeled as the composition of an invertible linear affine map and the flow of a learned, time-invariant nonlinear vector field, trained with a two-stage scheme. The authors state this is the first demonstration that a vector-field mismatch loss (which contains the product of the Jacobian of the model and the vector field under comparison) can be minimized end-to-end, in contrast to scalar-field registration losses that only involve a Jacobian determinant.

3. A continuous similarity index for topologically distinct systems. For systems that are not equivalent, DFORM output generalizes binary orbital equivalence into a graded measure, the orbital similarity, defined as the minimum of forward and backward alignment.

4. A template-matching route to dynamical motif identification. By aligning a high-dimensional system to a low-dimensional template, DFORM can locate invariant manifolds and saddle limit sets (including saddle limit cycles with unknown period) without assuming asymptotic stability.

Main Findings

• Linear transformations are recovered even at high dimension. For the four categories of topologically equivalent linear systems tested (orthogonal, general linear, same type, same sign), forward alignment between the pushforward of one system and the other remained above 0.95 even at n = 128 for linearly related systems, and was mostly above 0.8 for nonlinearly related systems, across 30 randomly generated pairs per category.

• Multiple optimal solutions exist. Although DFORM aligned linearly transformed systems well, the transformations it found were generally very different from the ground truth (Supplementary Figure S1). The authors prove that for a linear system, an arbitrary invertible polynomial function of the system matrix can be inserted to produce an identical pushforward vector field, so the ground-truth transformation can only be identified up to an equivalence class.

• Orbital similarity tracks signature concordance. In 16-dimensional linear systems with signatures (16,0,0), (12,4,0), (8,8,0), (4,12,0) and (0,16,0), mean orbital similarity was beyond 0.8 for topologically equivalent systems, and decreased gradually as signature concordance decreased, across 20 experiments for each of the 5² = 25 ordered pairs.

• Nonlinear systems align well under linear transformations. Using RNNs with "random plus low rank" connectivity, an orthogonal ground-truth transformation was recovered even at n = 128, with near-perfect cosine similarity of fixed points or Jacobian matrices; for non-orthogonal transformations, fixed-point alignment was above 0.9 in general. Suboptimal solutions were observed across initializations where Jacobian similarity was very high but fixed-point similarity was low.

• Low-dimensional motifs can be located. DFORM was demonstrated to locate invariant manifolds and saddle limit sets within high-dimensional models by projecting the vector field to a lower-dimensional template and mapping the located features back through the inverse transformation.

• Application to empirical neural data. Using a set of RNN models trained on human functional magnetic resonance imaging (fMRI) recordings, DFORM identified limit cycles from the high-dimensional data-driven models, and these agreed well with prior numerical analysis.

Methodology in Plain English

Two dynamical systems are called smoothly orbitally equivalent if a smooth, invertible coordinate change maps the trajectories of one exactly onto the trajectories of the other, one-to-one. Finding such a map by hand is intractable for anything but the simplest systems, so the authors train a neural network to find it.

The transformation is built in two parts: a linear affine map (which can translate, scale, rotate and reflect the state space) followed by the flow of a learned nonlinear vector field, implemented as a Neural ODE. Composing these two gives the diffeomorphism, and its inverse is obtained by running the flow backward in time.

Training minimizes a loss that compares the direction of the transformed vector field to the direction of the target vector field, ignoring magnitude. This direction-only comparison means DFORM is not sensitive to speed differences between systems by default, which lets it align systems that are topologically equivalent but not strictly diffeomorphic. The loss is written in both directions (one in each system's state space) and both terms are minimized, because the two terms are usually numerically different.

Implementation details as reported: a two-hidden-layer feedforward network learns the deformation field, with hidden layer width set to 2n or 20, whichever is larger, and ELU activations to guarantee smoothness; the flow is integrated with a Dormand–Prince–Shampine Runge–Kutta method of order 5 with relative tolerance 10⁻⁵; the linear map is a single n-to-n linear layer. Total parameter counts are n² + 42n + 440 for n < 10 and 9n² + 6n for n ≥ 10. Training proceeds in two phases: first the linear map alone at a NADAM learning rate of 0.002, then all weights at 0.0002. The largest experiment reported involved two 100-dimensional systems, 2000 batches of full-scale training, and about 20 minutes on an Nvidia V100 GPU.

Why This Matters

The paper attacks a practical obstacle in computational neuroscience: models fit to different subjects or sessions live in different coordinate systems, making mechanistic comparison unreliable. DFORM turns a rigorous but previously non-operational mathematical notion into a trainable procedure, and it provides a way to find features that forward simulation can never surface, since states converging to saddle limit sets form a set of measure zero.

Real-world applications (as framed by the paper or directly implied by it):

• Neuroscience model comparison: judging whether two RNNs trained on different data or tasks implement the same mechanism despite different coordinate systems.
• Analysis of brain imaging models: the authors applied DFORM to RNNs trained on human fMRI recordings to identify limit cycles, matching prior numerical analysis.
• Understanding metastable brain states: the paper cites saddle limit sets as mediators of transitions between metastable brain states, a feature DFORM can locate.
• Decision-making models: the paper cites winner-take-all mechanisms in decision-making as an example of dynamics built on improbable limit sets that DFORM can identify.

Industry relevance: The paper does not report industrial deployments, benchmarks, or commercial applications. Its stated context is theoretical neuroscience and scientific machine learning. Any industry relevance would be indirect, for example in comparing learned dynamical models across systems where coordinate systems are not shared, but the paper does not present such use cases.

Future Directions

• Resolving the non-uniqueness problem. Because multiple optimal transformations exist and the ground truth was generally not recovered in the linear experiments, the authors note this has important implications for nonlinear alignment, where linearization-based methods may fail because of the multiplicity of solutions. Identifying transformations only up to an equivalence class remains an open issue.

• Handling speed-sensitive alignment. DFORM is by default insensitive to speed differences between systems; the authors state it can be made sensitive if desired (see their Discussion), but the paper content provided does not detail results from doing so.

• Extensions to input-driven systems. The authors note that constant or piece-wise constant exogenous inputs are readily absorbed into the formulation of the vector fields, with further discussion deferred to their Discussion section.

• Scaling and further empirical validation. The largest experiment reported is two 100-dimensional systems; extending template matching and motif identification to larger, more heterogeneous empirical model sets is a natural next step.

Target Audience

Researchers in theoretical neuroscience and scientific machine learning who build or analyze recurrent neural network models and want to compare dynamics across models; dynamical systems researchers interested in operationalizing smooth orbital equivalence; and machine learning practitioners working with Neural ODEs, diffeomorphic transformations, or model alignment. The paper assumes familiarity with dynamical systems concepts such as limit sets, topological equivalence and Jacobians, so it is best suited to readers with an intermediate-to-advanced background.

Authors’ abstract

Dynamical systems models such as recurrent neural networks (RNNs) are increasingly popular in theoretical neuroscience for hypothesis-generation and data analysis. Evaluating the dynamics in such models is key to understanding their learned generative mechanisms. However, such evaluation is impeded by two major challenges: First, comparison of learned dynamics across models is difficult because there is no enforced equivalence of their coordinate systems. Second, identification of mechanistically important low-dimensional motifs (e.g., limit sets) is intractable in high-dimensional nonlinear models such as RNNs. Here, we propose a comprehensive framework to address these two issues, termed Diffeomorphic vector field alignment FOR learned Models (DFORM). DFORM learns a nonlinear coordinate transformation between the state spaces of two dynamical systems, which aligns their trajectories in a maximally one-to-one manner. In so doing, DFORM enables an assessment of whether two models exhibit topological equivalence, i.e., similar mechanisms despite differences in coordinate systems. A byproduct of this method is a means to locate dynamical motifs on low-dimensional manifolds embedded within higher-dimensional systems. We verified DFORM's ability to identify linear and nonlinear coordinate transformations using canonical topologically equivalent systems, RNNs, and systems related by nonlinear flows. DFORM was also shown to provide a quantification of similarity between topologically distinct systems. We then demonstrated that DFORM can locate important dynamical motifs including invariant manifolds and saddle limit sets within high-dimensional models. Finally, using a set of RNN models trained on human functional MRI (fMRI) recordings, we illustrated that DFORM can identify limit cycles from high-dimensional data-driven models, which agreed well with prior numerical analysis.