Predictive Coding Graphs are a Superset of Feedforward Neural Networks

AI summary

Overview

Research area: Machine learning theory and computational neuroscience, specifically predictive coding networks (PCNs) and predictive coding graphs (PCGs), and their formal relationship to feedforward neural networks (FNNs).

Technical level: Advanced. The paper is a purely theoretical work consisting of definitions, two theorems and their proofs (including an appendix of derivations and gradient update rules).

Scope: A formal proof that predictive coding graphs define a mathematical superset of feedforward artificial neural networks (multilayer perceptrons), established by proving two smaller results.

What This Paper Is About

Predictive coding networks are a neuroscience-inspired model that has drawn interest in machine learning as a more biologically plausible alternative to backpropagation (BP). Prior work had shown that standard hierarchical PCNs converge to the computations of an FNN during testing, and that PCGs could be shaped into hierarchical structures by masking weights, but the precise relationship between PCGs, PCNs and FNNs was not formally pinned down. This paper supplies those proofs, showing that a PCN is equivalent to an FNN at test time and that a PCG with a particular weight matrix is exactly equivalent to a PCN in both objective and dynamics.

Key Contributions

1. A proof that a PCN is equivalent to an FNN during testing — a stronger reframing of earlier results that only showed convergence. The author notes the proof is simpler than comparable proofs in the literature (Song 2021; Salvatori et al. 2022b; Frieder and Lukasiewicz 2022) because it uses a more general definition of the activity rule instead of assuming gradient-based dynamics.

2. A rigorous justification, stated by the author to be the first, that the universal approximation theorem (UAT, Cybenko 1989) applies to PCNs, obtained as a corollary of the PCN–FNN testing equivalence.

3. A formal proof that a PCG with the block-diagonal weight matrix setting w̃^{ℓk} = w̃^{ℓk} δ_{k ℓ−1} ≡ w^{ℓ−1} is exactly equivalent to a PCN: the energies match up to a constant, and the activity rules (training and testing) and learning rules are identical.

4. The combined conclusion that PCGs are a structural superset of FNNs, including FNNs as a special case, with a visualization of how skip connections appear as part of the PCG weight matrix (the paper states this extends Figure 4 in Salvatori et al. 2022a).

Main Findings

• Theorem 1 — PCNs equal FNNs at test time. During testing, the PCN activity rule argmin over a_i^ℓ of E_N is equivalent to the FNN's forward pass a_i^ℓ = f(Σ_j w_{ij}^{ℓ−1} a_j^{ℓ−1}). The proof solves the stationary conditions ∂E_N/∂a_i^ℓ = 0 by backwards induction: since ε_i^L = 0, equation (6) gives ε_i^{L−1} = 0, and this continues down to ε_i^1 = 0.

• Universal approximation carries over. Because FNNs are universal function approximators, so are PCNs. The paper describes this as a fact probably widely believed in the predictive coding community but previously lacking a rigorous justification.

• Theorem 2 — PCGs contain PCNs. Partitioning N PCG nodes into layers {n_ℓ} with Σ_ℓ n_ℓ = N and the weight matrix w̃ into block matrices w̃^{ℓk} ∈ R^{n_ℓ × n_k}, choosing the hierarchical structure yields E_G = E_N + C, where C is a constant, and identical activity rules in training and testing and identical learning rules in training.

• The constant. C is a term from the input layer and is constant because the lowest layer is clamped to the data during both training and testing. For the "Matrix-Activation" convention C = Σ_i [a_i^0 − f(0)]²; for the alternative "Activation-Matrix" convention C = Σ_i (a_i^0)², since the non-linearities are multiplied by zero.

• What other weight matrices give. Non-zero block matrices other than the hierarchical choice produce skip connections, backward (skip) connections and lateral connections.

• Corollary for PCGs. During testing, PCGs are also a superset of FNNs (by Theorem 1), and are therefore universal function approximators when the hierarchical structure of equation (15) is chosen. Whether PCGs with additional connections can approximate any function is explicitly left for future work.

• Computational cost. For an FNN, testing one batch is O(LM), where M = max({n^ℓ n^{ℓ+1}}_ℓ) is the number of weights in the largest block matrix of w̃. For a PCG it is O(N²T), where T is the number of inference steps, reducing to O(dNT) if sparsity is exploited, with d the number of non-zero weights. Since d ≈ LM for an FNN with a comparable number of weights, testing one batch is a factor NT slower in a PCG. The paper argues this is not necessarily undesirable.

• Prior empirical results cited. Only all-to-all connected PCGs (the full matrix except self-connections) have been studied empirically (Salvatori et al. 2022a). The paper reports that these appear not to perform as well as hierarchical PCNs/FNNs but outperform other all-to-all connected networks for classification on MNIST by a large margin — 12-35% better than Boltzmann machines and Hopfield networks for three datasets. No new experiments are reported in this paper.

• Recurrence is different from RNNs. The non-feedforward connections introduce a notion of recurrence in "inference time", like Hopfield networks, rather than recurrence with respect to "data time" characterized by weight sharing as in recurrent neural networks. The paper states this distinction was not highlighted in Salvatori et al. (2022a).

• Initialization matters in practice. PCNs perform iterative gradient-based minimization of E_N with respect to non-clamped activations, so the result of inference depends on initialization. A feedforward pass initialization is typically used in the literature and appears to give better performance than random or zero initialization (Pinchetti et al. 2024). This scheme depends on a feedforward structure, and an analogous scheme for a PCG with both forward and non-forward connections is stated to be not obvious and would have to be separately implemented.

Methodology in Plain English

The paper is a mathematical argument rather than an experimental study. The approach has three parts.

First, the author separates a network's activity rule (how node activations change, operating during both training and testing) from its learning rule (how weights change, operating only during training), following MacKay (2003). For FNNs the learning rule is normally backpropagation; for PCNs the learning rule is treated as part of the network definition itself.

Second, the author writes down competing definitions of an FNN, a PCN and a PCG, each built around a squared-prediction-error energy — E_N for the PCN, E_G for the PCG. To show PCN–FNN equivalence, the paper differentiates the PCN energy with respect to each node activation, sets the derivative to zero, and then uses backwards induction starting from the output layer, where ε_i^L = 0, to show every error term must vanish, which recovers exactly the FNN forward pass.

Third, to show that PCGs contain PCNs, the author defines an explicit index mapping between the two formalisms: PCG node α belongs to layer ℓ if α is in the index set I_ℓ = {s_{ℓ−1}+1, …, s_ℓ}, where s_ℓ = Σ_{k=0}^{ℓ} n_k, and the corresponding PCN index is i = α − s_ℓ. With that mapping, the PCG weight matrix is partitioned into block matrices, the PCG energy is decomposed layer by layer, and substituting the mapping shows the two energies differ only by a term that is constant because the input layer is always clamped to the data.

Why This Matters

The paper sharpens a relationship that was previously described only loosely. Saying that PCNs converge to an FNN's computation leaves room for approximation error; saying they are equivalent at test time allows the universal approximation theorem — historically a key theoretical justification for using FNNs — to be applied to PCNs as well. It also makes concrete what a PCG adds: not just a different way of drawing a network, but a strictly larger space of structures that includes FNNs exactly, and that includes skip, backward, lateral and self-connections that backpropagation cannot train in the paper's framing.

The paper presents no experiments of its own, so the applied consequences below are areas its framing speaks to rather than results it demonstrates.

• Biologically plausible learning. The paper positions PCGs as trainable with inference learning (IL), described as a more biologically plausible alternative to BP, which is relevant to research that studies learning rules that could operate in biological tissue.
• Probabilistic and generative modeling. The paper cites PCN potential for probabilistic/generative modeling as part of the motivation, so the superset result carries over to that line of work.
• Parallelizable training. The paper cites the parallelizability of PCNs (Salvatori et al. 2024) as a motivating property, which is relevant to hardware and distributed-training research.
• Architecture and topology research. The paper argues that since skip connections — the main innovation in ResNets (He et al. 2016), which the paper notes are well known to benefit many ML tasks compared to standard MLPs — appear as entries of the PCG weight matrix, the framework gives a principled way to study the value of network topology, and by analogy to ask whether backward, lateral and self-connections bring similar benefits.

Industry relevance: the paper does not report industrial deployments or benchmarks. Its relevance to practice is indirect: it clarifies a training alternative to backpropagation, quantifies the testing-time cost of using non-feedforward PCG structures (a factor NT slower per batch, or a factor N T relative to a comparable FNN), and notes that PCN performance depends on initialization in a way that a non-hierarchical PCG cannot trivially replicate.

Future Directions

1. Function approximation with non-hierarchical PCGs. The paper explicitly leaves open whether PCGs with connections beyond the hierarchical structure can also approximate any function.
2. Empirical study of the remaining connections. The paper asks whether the rest of the PCG weight matrix — backward (skip) connections, lateral connections and self-connections — brings the same kind of benefit that skip connections bring in ResNets. It notes that only all-to-all connected PCGs have been studied empirically so far.
3. Initialization for non-feedforward PCGs. Since the standard feedforward-pass initialization depends on having a feedforward structure, a scheme for PCGs with forward and non-forward connections must be developed separately.
4. Mitigating the cost of non-feedforward connections. The paper identifies gradient-based inference of nodes as relatively expensive and notes that the O(N²T) testing cost can drop to O(dNT) when sparsity is exploited, pointing to efficiency as an open practical problem. It also notes that the paper's theoretical style is complementary to the experimental approach more common in the literature and calls for more rigorous mathematical study of PCNs (echoing Frieder et al. 2024 and van Zwol et al. 2024).

Target Audience

This paper is aimed at machine learning theorists, predictive coding researchers and computational neuroscientists who want a precise account of how predictive coding models relate to ordinary feedforward networks. It will also be useful to engineers and researchers working on biologically plausible learning rules, alternative training algorithms to backpropagation, and neural architecture/topology research, provided they are comfortable with energy-based formulations and inductive proofs. Readers looking for benchmark results or implementation guidance will find only the cost analysis, the initialization-convention caveat and the practical note that weights must be explicitly set to zero according to the block structure to keep a desired topology.

Authors’ abstract

Predictive coding graphs (PCGs) are a recently introduced generalization to predictive coding networks, a neuroscience-inspired probabilistic latent variable model. Here, we prove how PCGs define a mathematical superset of feedforward artificial neural networks (multilayer perceptrons). This positions PCNs more strongly within contemporary machine learning (ML), and reinforces earlier proposals to study the use of non-hierarchical neural networks for ML tasks, and more generally the notion of topology in neural networks.