Architecture Before the Formula: Individuating Neural Architecture Beyond the Composite Map

AI summary

Overview

Research area: Machine learning theory — specifically the foundations of neural architecture search, architecture identity, and representation/composition analysis.

Technical level: Advanced. The paper is a formal, theorem-and-proof treatment built on set maps, kernels (equivalence relations), surjective factorizations, direct-sum vector spaces, and affine maps, with connections drawn to information order (Blackwell), network bisimulation, and categorical deep learning. There are no experiments, datasets, or training runs.

Scope (one sentence): The paper argues that neural architecture should be individuated at the level of the represented receiver process — the actual intermediate state a branch supplies for further computation — rather than by module syntax labels, computation graphs, or the composite function alone.

What This Paper Is About

When we say two architectures are "the same," we usually mean they share module names, graph structure, or the overall input-to-output function they compute. The paper points out that these are different identity questions, and that the composite function B_j = G_j Q_j has many different intermediate presentations (G_j Q_j = (G_j T^{-1})(T Q_j) for any bijection T). The goal is to give an exact account of what survives when you deliberately forget the intermediate presentation and keep only which predecessor inputs become indistinguishable at a selected architectural cut — and then to show that downstream architectural distinctions can depend on how upstream computation produced its state.

Key Contributions

1. Architecture is finer than its extensional shadow. The paper keeps the represented interface/continuation process before quotienting by predecessor-state distinctions. For fixed-branch surjective factorizations, the "distinction shadow" (the kernel of the interface map) exactly classifies the unmarked factorization class up to a unique carrier bijection — but marked roles identify structure that this quotient forgets.

2. Recoverable information and represented accessibility are separate questions. Equal distinction shadows guarantee a unique carrier conversion, but that conversion need not preserve marked organization (tokens, heads, sites, sources) or the continuation family available at the receiver. Deterministic and squared-loss results separate information lost at the interface from limits imposed by how surviving information is presented and used.

3. Downstream architectural identity can depend on upstream representation. "Recutting" evaluates downstream interfaces after the prefix that produces their inputs. The paper reports an exact two-token construction in which local and attention schemas collapse to the same distinction-class set after one injective prefix and separate after another — even though neither prefix loses predecessor information, removing the ordinary information-loss explanation.

4. A formal toolkit for exact architecture comparison. Marked receiver presentations, mark signatures, admissible mark-transport systems, marked locality, affine locality by block support, and a continuation-transport theorem are introduced to make the architecture question precise.

Main Findings

• Composite functions do not determine intermediate presentations. Lemma 4 (Factorization contraction): for B = GQ and any bijection T: Z → Z', B = (GT^{-1})(TQ). If T is a passive redescription, the incident maps and marks must be transported with the chart; if T is an arrow in the represented process, it is part of the computation.

• The distinction shadow exactly classifies unmarked factorizations. Theorem 11 shows Q ≡_D Q' (equal kernels) if and only if there exists a unique bijection T with Q' = TQ. Theorem 12 extends this: for a fixed branch, Q ≡_D Q' iff the surjective factorizations are isomorphic as unmarked factorizations, and the isomorphism is unique.

• Equal shadows do not imply equal marked processes. The paper states that for fixed-branch surjective factorizations, the shadow classifies the unmarked class exactly but does not determine marked receiver organization or the accessibility of retained information to restricted continuations. The admissible mark-transport system is fixed before a candidate carrier conversion is tested.

• Explicit counterexample (Example 26). With X = Z = Z' = ℝ², aligned scalar-coordinate marks, Q(u,v) = (u,v), Q'(u,v) = (u+v, u−v) via T = [[1,1],[1,−1]], and common branch B(u,v) = u with G(a,b) = a and G'(s,d) = (s+d)/2: the shadows are equal and the unmarked factorizations are isomorphic, but T is not block diagonal, so the marked receiver processes are not equivalent up to passive marked re-presentation. Recovering u requires the cross-coordinate continuation (s,d) ↦ (s+d)/2 rather than a one-coordinate readout.

• Invertible affine mixing can change restricted accessibility (Example 32). With the same T, if the allowed continuation class is exactly the one-coordinate functions, the target f(u,v) = u is attainable from Q but not from Q', even though Q ≡_D Q'.

• Aggregation cannot improve target resolution. Theorem 28 (Deterministic distinction barrier): for any interface E and continuation G, sup_x d(f(x), G(E(x))) ≥ (1/2) Δ_E(f), and an exact continuation exists iff Δ_E(f) = 0. Proposition 29 shows Δ_E(f) ≥ Δ_P(f) when E = Agg P — every P-fiber sits inside an E-fiber. These statements are invariant under injective post-processing.

• Affine marked structure has a clean characterization. Theorem 18: an affine map on direct sums is local with neighborhoods (N_r) iff every block A_{ri} = 0 for i ∉ N_r. Corollary 25: an affine mark-preserving bijection has block-diagonal invertible linear part, and a block-monomial one when an explicit receiver permutation is admitted.

• Continuation transport is exact. Theorem 31: if Q' = TQ for a bijection T, then attainable branch families satisfy 𝔅(Q', 𝒢') = 𝔅(Q, T*𝒢'), and because Q is surjective, 𝔅(Q, 𝒢) = 𝔅(Q', 𝒢') iff 𝒢 = T*𝒢'.

• Context dependence is hidden by modular labels. The paper reports that kernel composition satisfies ker(Q_{j,θ}A) = (A × A)^{-1} ker Q_{j,θ}, so a downstream module label need not name an independent architectural degree of freedom once the upstream representation is fixed.

• What is not reported. The truncation of the paper content means the full two-token construction, the Transformer attention and PreNorm block illustrations, and any further lossy-construction results referenced from the contributions list are not visible in the supplied text. No datasets, training results, benchmarks, or empirical comparisons are reported anywhere in the available content.

Methodology in Plain English

The paper works entirely at the level of mathematics, not experiments. It fixes a represented update F: X → Y and a receiver branch B_j: X → W_j, then factors the branch as B_j = G_j Q_j, where Q_j(x) is the state the architecture supplies at a chosen cut and G_j is the computation that follows. The key move is to deliberately throw away the coordinates of Q_j(x) and keep only its kernel — the set of predecessor input pairs that Q_j maps to the same value. This kernel is called the distinction shadow.

The paper then proves a series of small, self-contained lemmas and theorems. Some characterize the shadow (it commutes with precomposition, intersects under joint presentation, and is unchanged by injective post-processing). Others classify factorizations (equal shadows give a unique carrier bijection). Still others add marks — declared maps recording roles like token, site, head, or source — and define a fixed set of admissible mark transports that say which relabelings count as passive. A marked process is then compared only up to those admitted transports. Finally, the analysis is pushed through composition: the downstream interface is Q_{j,θ}A, whose shadow is computed from the upstream map A and the downstream kernel, and a barrier theorem bounds how well any continuation can match a target given a fixed interface.

The result is a vocabulary and a set of exact statements — not a new training algorithm.

Why This Matters

Impact on research. The paper challenges a common informal assumption in architecture search and representation comparison: that comparing realized functions, graph structures, or module labels is enough. It shows that "same information" and "same represented organization" are different claims, and that a downstream architectural choice can be distinguishable or indistinguishable depending on a property of the upstream prefix that neither choice names. It also connects to existing lines of work — graph-isomorphism quotients in NAS-Bench-101 (Ying et al., 2019), encoding-dependent search behavior (White et al., 2020), network morphisms (Wei et al., 2016), attention realizing convolutions (Cordonnier et al., 2020), representation similarity (Kornblith et al., 2019), identifiability up to linear indeterminacy (Roeder et al., 2021), comparison-of-experiments orders (Blackwell, 1951), network bisimulation (Prabhakar, 2022), message passing and graph networks (Gilmer et al., 2017; Battaglia et al., 2018), categorical deep learning (Gavranović et al., 2024), interface theories (Tripakis et al., 2011; Broy, 1997), and causal abstraction (Geiger et al., 2022).

Real-world applications (the paper does not name applications; these are the areas its concepts bear on):

• Architecture search and evaluation: deciding when two candidate cells or branches should be counted as the same search-space point.
• Model comparison and reproducibility: reporting whether two checkpoints differ in information or only in how that information is presented and used.
• Compression, refactoring, and conversion tooling: distinguishing passive chart changes from transformations that add or remove represented computation.
• Mechanistic interpretability and probing: separating what a probe can recover from what the model's own continuation can actually read out.

Industry relevance. Practitioners routinely treat functionally equivalent models as interchangeable and treat module labels as architectural degrees of freedom. The paper's results imply that an invertible change of intermediate presentation can leave every predecessor distinction intact while breaking a restricted readout, which bears on distillation, weight conversion, layer merging, and prompt- or head-level ablations where the downstream computation is constrained (for example, to a single coordinate or a fixed head).

Future Directions

• Restricted-continuation theory. The paper separates information lost at the interface from limits imposed by presentation and use; developing a general account of which continuation families are admissible, and when a carrier conversion is a passive re-presentation rather than an active arrow, is the obvious next step.

• Beyond exact shadows. The analysis uses exact kernel equality. A quantitative analogue for stochastic or approximate interfaces — building on the Δ_T(f) variation measure and the deterministic barrier — is not developed in the available content.

• Scaling the two-token witness. The reported equivalence/inequivalence of local and attention schemas after two different injective prefixes is a minimal construction; extending it to realistic Transformer attention and PreNorm blocks (mentioned but not shown in the supplied text) would test how far the phenomenon reaches.

• Where the comparison stops. The paper states it stops short of complete implementation identity and causal-mechanism identity, leaving open how the represented-process criterion should be composed with intervention-based criteria such as causal abstraction.

Target Audience

Researchers and graduate students in machine learning theory, neural architecture search, representation learning, and interpretability who want a rigorous vocabulary for asking when two architectures are the same. It will also interest readers from programming languages, verification, and systems who work on interface refinement and substitutability, since several of the tools (mark signatures, admissible transports, localization by support) have direct analogues there. Practitioners looking for empirical guidance or benchmark results will not find them in this paper.

Authors’ abstract

Neural architecture is often identified by module syntax, computation graphs, or the composite functions they realize. These descriptions answer different identity questions. We study the represented process available at a receiver: an actual factorization $B_j=G_jQ_j$ in which $Q_j(x)$ is the intermediate state supplied for further computation. Forgetting the presentation and retaining only $\ker Q_j$ yields the predecessor distinctions preserved at that cut. For a fixed branch, this extensional shadow exactly classifies unmarked surjective factorizations up to unique carrier re-presentation, but it does not determine marked receiver organization or the accessibility of retained information to restricted continuations. Under composition the relevant interface is $Q_{j,θ}A$, so downstream architectural distinctions depend on the states produced upstream. An exact two-token construction shows that local and attention schemas are distinction-equivalent after one injective prefix and inequivalent after another, even though neither prefix loses predecessor information. The result exposes a context dependence hidden by modular architecture labels and motivates architecture comparison at the represented-process level.