Modularity is the Bedrock of Natural and Artificial Intelligence

AI summary

Overview

Research area: Artificial intelligence and neuroscience — specifically the role of modularity as a design and organizational principle in both biological and artificial systems. The paper pulls in evidence from evolutionary biology, complex systems, and engineering theory as well.

Technical level: Advanced. The paper is a conceptual review, but it presents a formal definition of modular models (routing, aggregation, and modifier functions) and synthesizes a large body of technical literature across several fields.

Scope in one sentence: A survey arguing that modularity is not merely a recurring feature of different scientific domains but a fundamental computational principle shared by natural and artificial intelligence, and therefore a promising core design principle for future AI systems.

Note on completeness: the supplied content of the paper ends mid-sentence in Section 3.3.3 (on Mixture-of-Experts layers), so the detailed contents of Sections 4 (brain modularity), 5 (brain-inspired AI context), 6 (open questions), and 7 (takeaways) are not available in the text provided here.

What This Paper Is About

Modern AI systems achieve impressive results but require data, computation, and energy that vastly exceed what human intelligence needs — for example, tens of trillions of tokens for current LLMs and an estimated 1287 MWh to train GPT-3. The author argues that the No Free Lunch Theorem implies no single monolithic architecture can be a universal problem solver, so good inductive biases must be problem-specific, which favors architectures built from specialized modules. The goal is to review and synthesize research across AI, neuroscience, evolutionary biology, complex systems, and engineering to show that modularity underpins both natural and artificial intelligence, and to argue it should be a central design principle for future AI.

Key Contributions

1. A cross-disciplinary conceptual framework. The paper surveys research threads from artificial intelligence, neuroscience, evolutionary biology, complex systems, and engineering theory, and argues that modularity appears in each but has been studied in isolation, becoming associated with different properties and functions in each field (for example, robustness and communication efficiency in complex systems; evolvability and adaptability in evolutionary biology; energy efficiency and functional specialization in neuroscience).

2. A proposed taxonomy of modularity in AI. Table 2 organizes modularity into three categories — implicit modularity (units in neural networks, layers in deep neural networks, winning tickets in the Lottery Ticket Hypothesis), emergent modularity (functional modules in multi-task learning, knowledge neurons in LLMs, induction head circuits in LLMs), and architectural modularity (experts in Mixture-of-Experts, retrieval-augmented generation databases in industrial chatbots, agents in multi-agent systems).

3. A formal definition of modular models. The paper defines a model as modular if it can be written as f(x) = φ(m_μ1(x), m_μ2(x), …, m_μM(x)), and introduces a routing function r_ρ(x): ℝ^i → 2^ℳ and an aggregation function g_γ(x): 2^ℳ → ℝ^o, giving the compact form f(x) = g_γ(r_ρ(x)). It also distinguishes hard routing (sparse, efficient at inference, but not trainable end-to-end by gradient descent) from soft routing (trainable end-to-end but not sparse), and introduces a modifier function d_δ^(f) for modules used to fine-tune pre-trained models.

4. A synthesis of the computational advantages of modularity. The paper reviews properties such as information factorization, partial autonomy, amortized control, and timescale separation, and connects them to evidence in evolutionary simulations, artificial neural networks, and the human motor system.

5. An argument for brain-inspired module design. The author proposes that because modular architectures are hard to design effectively, insights from the brain may help identify the fundamental functions that modules should specialize in, offering an avenue to bridge natural and artificial intelligence.

Main Findings

• Efficiency gap between AI and human intelligence. CNNs reach high classification accuracy only after training on thousands of examples per class, whereas humans can learn from a single example. AlphaGo achieved superhuman performance only after training on at least three orders of magnitude more games than elite human players. Current LLMs require datasets of tens of trillions of tokens, which would take an above-average reader 5–50 thousand years of continuous reading.

• Energy disparity. Training GPT-3 was estimated to consume 1287 MWh — enough to power over 100 average American households for a year, and roughly three orders of magnitude more than the 3.15 MWh required to power a 20W human brain for 18 years. The paper notes that more recent models are rumored to have at least 10X more parameters and to rely heavily on additional test-time computation.

• Persistent limitations of current AI. Despite their resource use, current systems still struggle with out-of-distribution generalization, have poor compositional skills, and suffer from catastrophic forgetting and limited positive transfer — limitations humans typically do not share.

• Modularity in engineering and nature. Modular systems are defined as composed of subsystems whose elements are strongly connected internally and weakly connected across subsystems, implementing divide-and-conquer through information factorization. Baldwin and Clark (1999) identified six modularity operators: splitting, substituting, augmenting, excluding, inverting, and porting. Most complex systems — biological, physical, social, symbolic — are widely regarded as modular, often hierarchically, and graph-theoretical analyses often reveal small-world and modular organization with dense local interactions, long-range shortcuts, and connector hubs.

• Evolutionary explanations of modularity. Simulation studies suggest modularity arises in response to changing environments, is particularly strong when the environment changes in a modular manner (Kashtan and Alon, 2005), and speeds up adaptation to new environments (Kashtan et al., 2007). Recent studies found multi-task learning promotes modularity in CNNs and RNNs, and minimization of connection costs in constant environments also leads to modular solutions (Clune et al., 2013).

• Computational advantages of modular brain networks. Networks biased toward short connections decompose tasks into subtasks, show greater resilience to catastrophic forgetting in continual learning, demonstrate brain-like mixed selectivity and low average activation patterns, and form sparse information streams while reusing useful features. Key named advantages include information factorization, partial autonomy, amortized control, and timescale separation.

• Implicit modularity in deep networks. DNNs are stacked non-linear layers — a hierarchical modular architecture. Poggio et al. (2017) narrowed the class of functions DNNs excel at to compositional ones of the form f(x) = h_L(h_{L−1}(…h_1(x))). For compositional functions of n = 8 variables with smoothness m and a binary tree-like computational graph, a shallow network needs O(ε^{−n/m}) units, while a deep network needs O((n−1)ε^{−2/m}) units. Convolutional layer units act as feature detectors, and winning tickets from the Lottery Ticket Hypothesis are described as specialized subnetworks that can serve as modules; overparameterized random DNNs contain multiple such tickets capable of solving a task with comparable accuracy to the full dense network even without additional training.

• Emergent modularity and its limits. Modules are traditionally identified by clustering connectivity or activation statistics. Binary-mask approaches identified subfunction-specific subnetworks, but follow-up analyses showed these subnetworks were not reused in different contexts where the same rules had to be applied in new combinations — evidence of a lack of compositionality, related to the problem of information binding. Mechanistic interpretability has identified specialized components such as knowledge neurons and induction head circuits, and such components can endow models with a degree of compositional ability, especially at larger scales.

• Architectural modularity depends on known task structure. Modular architectures show better generalization and sample efficiency than monolithic counterparts, but strong compositional generalization is observed only when the task structure is known and can be used to assign modules to subtasks. When the task structure is unknown, modular networks often do not specialize and show no consistent performance boost. Bahdanau et al. (2018) found a perfect one-to-one mapping between subtasks and modules is not always best; Béna and Goodman (2021) found strong inter-module connection sparsity and resource constraints facilitate specialization.

• Continual learning, transfer learning, and LLMs. Task-specific parameters trained only with task-specific signals cannot suffer interference by construction. Approaches include freezing a shared network and adding fixed-capacity modules (Rusu et al., 2016), selectively fine-tuning shared parameters with module capacity depending on task difference (Yoon et al., 2017), and selecting and freezing task-specific paths through a fixed high-capacity network (Fernando et al., 2017). In transfer learning, training only the last layers or small task-specific adapter modules is typically sufficient rather than fine-tuning the entire network, and this supports multi-task cross-lingual transfer by combining task- and language-specific adapters. Model MoErging composes independently pre-trained expert models, and Augmented Language Models combine LLMs with task-specialized modules such as a calculator, a retriever, a web browser, or a code interpreter.

Methodology in Plain English

This is a review and synthesis paper rather than an experimental study; the author reports no new experiments, benchmarks, or datasets. The approach is:

• Literature synthesis across fields. The author surveys and connects research from artificial intelligence, neuroscience, evolutionary biology, complex systems, and engineering theory, showing where modularity has been studied separately and how the findings relate.
• Conceptual framing. The paper introduces modularity as a shared computational principle rather than a domain-specific feature, and uses the No Free Lunch Theorem to argue that problem-specific inductive biases — and thus specialized modules — are necessary.
• Formalization. Section 2.3 gives a mathematical definition of a modular model with routing, aggregation, and modifier functions, then uses that notation to organize and compare existing modular architectural approaches.
• Taxonomy construction. Table 2 classifies modularity in AI into implicit, emergent, and architectural categories, with representative examples from the literature for each.
• Cross-domain comparison. Table 1 maps the properties and capabilities modular architectures are theorized to promote in each scientific domain, and the paper walks through modularity in the brain at spatial scales ranging from neurons and circuits to cortical networks, then contrasts these with AI systems.

Why This Matters

• Research impact. The paper argues that modularity deserves more attention in mainstream AI research and proposes it as a unifying lens connecting work that is currently scattered across continual learning, transfer learning, LLM design, reinforcement learning, autonomous agents, and mechanistic interpretability. It also proposes that neuroscience can guide what modules should specialize in, since designing modular architectures effectively remains difficult.

• Real-world applications:

  • Efficient fine-tuning and multi-task systems: adapter modules interspersed in pre-trained networks allow reuse of large models across domains and languages while minimizing fine-tuning costs.
  • Continual-learning systems: task-specific modules added to a shared network avoid interference by construction, which matters for systems that must acquire new tasks over time.
  • LLM scaling and composition: Mixture-of-Experts layers and Model MoErging compose independently trained expert models to expand capacity efficiently.
  • Tool-using assistants: Augmented Language Models combine LLMs with specialized modules for arithmetic, retrieval, web search, and code execution.
  • Motor control and robotics inspiration: the human motor system's spinal, cerebellar, basal ganglia, and cortical modules offer a template for reducing the dimensionality of control problems.

• Industry relevance. The efficiency gap the paper highlights is a direct cost problem: training GPT-3 was estimated at 1287 MWh, and next-generation LLMs with potentially at least 10X more parameters and heavy test-time computation are expected to increase energy consumption and carbon footprint further. Modularity is framed as a route to capacity and capability gains without proportional increases in training data, computation, and energy.

Future Directions

• Automatic discovery of latent task structure. Since modular networks only specialize when the task structure is known, the paper identifies a need for inductive biases and learning algorithms that can automatically discover latent task structure and filter out non-compositional features.

• Better module granularity and connection design. Open questions follow from findings that a perfect one-to-one mapping between subtasks and modules is not always best — architectures with modules specializing in groups of related subtasks can perform better — and that strong inter-module connection sparsity and resource constraints facilitate specialization.

• Learning compositional reuse. Since binary-mask studies showed identified subnetworks were not reused where the same rules applied in new combinations, understanding and overcoming the lack of compositional reuse and the related information binding problem remains open.

• Bridging natural and artificial intelligence. The paper's outline indicates discussion of open questions around the continued relevance of brains for developing AI systems, and the role of modularity in bridging the gap — specifically, using brain insights to identify the fundamental functions modules should specialize in. The detailed content of these sections is not included in the supplied text.

• A unified view of modularity across scales. The paper leaves open how the modularity principles observed at the levels of neurons, circuits, and cortical networks map onto and can inform the design of artificial modular architectures.

Target Audience

• AI researchers and architects interested in inductive biases, inductive bias design, modular architectures, Mixture-of-Experts, adapters, continual learning, and transfer learning.
• Computational and cognitive neuroscientists interested in how brain modularity at the neuronal, circuit, and cortical-network levels relates to computational efficiency and to AI design.
• Interpretability researchers working on emergent structure, knowledge neurons, and induction heads in large language models.
• Researchers in brain-inspired AI and cognitive science who want a cross-domain synthesis connecting evolutionary biology, complex systems, and engineering theory to machine learning.
• Graduate students and advanced practitioners looking for a conceptual map of why modularity appears across so many subfields and where the unresolved questions lie.

Authors’ abstract

The remarkable performance of modern AI systems has been driven by unprecedented scales of data, computation, and energy -- far exceeding the resources required by human intelligence. This disparity highlights the need for new guiding principles and motivates drawing inspiration from the fundamental organizational principles of brain computation. Among these principles, modularity has been shown to be critical for supporting the efficient learning and strong generalization abilities consistently exhibited by humans. Furthermore, modularity aligns well with the No Free Lunch Theorem, which highlights the need for problem-specific inductive biases and motivates architectures composed of specialized components that solve subproblems. However, despite its fundamental role in natural intelligence and its demonstrated benefits across a range of seemingly disparate AI subfields, modularity remains relatively underappreciated in mainstream AI research. In this work, we review several research threads in artificial intelligence and neuroscience through a conceptual framework that highlights the central role of modularity in supporting both artificial and natural intelligence. In particular, we examine what computational advantages modularity provides, how it has emerged as a solution across several AI research areas, which modularity principles the brain exploits, and how modularity can help bridge the gap between natural and artificial intelligence.