Neural networks
The Forward Pass and Computational Graphs
Trace forward computation through sequential, branching, shared, and stochastic operations while distinguishing values from the graph that produces them.
By the end you can
- Trace tensors through a forward pass in execution order
- Represent branches, merges, and reused values as a computational graph
- Distinguish deterministic operations from training-time stochastic behavior
- Identify values that must be retained for backward computation
Steps
A forward pass through a small residual block
Follow one input tensor rather than treating a model call as a black box.
The block traced here is not a generic sketch. It is the residual block, and written out it is y = F(x, {Wi}) + x. The input x travels two ways at once: forward through the weighted transformations that compute F, and around them untouched. The final step is where the two meet. Kaiming He and three colleagues at Microsoft Research published the design in 2016, and they describe that step in one line: “The operation F + x is performed by a shortcut connection and element-wise addition.” Nothing is added at the merge — no parameters at all. A projection Ws enters only when the input and output dimensions differ and the two tensors could not otherwise be added element by element.
The outcome of tracing exactly this block is on the record rather than in the abstract. An ensemble of these networks reached 3.57% top-5 error on the ImageNet test set. The official ILSVRC2015 results table lists MSRA's winning classification error as 0.03567.
1. Project
Apply an affine or convolutional transformation to the input.
2. Normalize
Rescale activations according to the block design.
3. Activate
Introduce a nonlinear response.
4. Transform again
Produce a residual update with the required output shape.
5. Add shortcut
Merge the update with the original or projected input.
The graph records dependencies, not a picture of neurons
A computational graph represents operations and the values that flow between them. Directed edges show which earlier results an operation needs. TensorFlow's own guide gives the parts exact names: “Graphs are data structures that contain a set of tf.Operation objects, which represent units of computation; and tf.Tensor objects, which represent the units of data that flow between operations.” Because they are data structures, they can be saved, run and restored without the original Python code.
Branches allow several transformations of one value. Merge operations such as addition or concatenation determine how those paths rejoin.
Whether that graph exists before the program runs is a design decision, and both sides of it are on paper. TensorFlow builds it first. Abadi and 21 co-authors at Google Brain stated the design in one line in 2016: “TensorFlow uses a unified dataflow graph to represent both the computation in an algorithm and the state on which the algorithm operates.” Computation and state, in a single structure, standing before any batch arrives.
Paszke and twenty co-authors described that older approach from the outside in 2019. Such frameworks “construct a static dataflow graph that represents the computation and which can then be applied repeatedly to batches of data”. There is a price. That approach “comes at the cost of ease of use, ease of debugging, and flexibility of the types of computation that can be represented”. PyTorch instead “performs immediate execution of dynamic tensor computations with automatic differentiation”. That design is called define-by-run. Under it the graph is a record of the execution that just happened. It is not a plan for one that has not.
The backward pass follows dependency structure created during forward computation.
Visual
Four graph patterns that appear everywhere
Recognizing these motifs makes unfamiliar architectures easier to read.
Branch and merge are not abstractions waiting for an example. The Inception module is built out of exactly that pair: one value feeds parallel 1×1, 3×3 and 5×5 filters plus a pooling path. Szegedy and eight co-authors state the rejoin: “It also means that the suggested architecture is a combination of all those layers with their output filter banks concatenated into a single output vector forming the input of the next stage.” The merge here is concatenation, not addition. The paths keep their identities and are laid side by side.
The network assembled from these modules is 22 layers deep, and the paper reports what it did: “Our final submission to the challenge obtains a top-5 error of 6.67% on both the validation and testing data, ranking the first among other participants.” The official ILSVRC2014 classification table records the winning GoogLeNet entry at 0.06656. Branch-and-concatenate did not become a standard pattern by argument.
Chain
Each operation consumes the previous result.
Branch
One value feeds several parallel computations.
Merge
Addition, concatenation, gating, or pooling combines paths.
Reuse
A parameter or activation participates in more than one location.
Comparison
Graph structure and runtime values are different objects
The same architecture executes many times. Each input still produces different activations.
Which of the two dominates memory is not a matter of impression. It was measured. Reverse-mode differentiation buys cheap derivatives with storage, and Baydin and three co-authors put that cost plainly in the Journal of Machine Learning Research in 2018: “The advantages of reverse mode AD, however, come with the cost of increased storage requirements growing (in the worst case) in proportion to the number of operations in the evaluated function.” Arithmetic is the cheap side of the same bargain. The same survey notes that “AD guarantees that the amount of arithmetic goes up by no more than a small constant factor”.
Read the two together and the asymmetry is the whole point. Compute grows by a constant factor. Storage grows with the number of operations the execution actually performed. The graph is fixed and small. The values it produced for one input are what scale.
Graph or program
Specifies operations, connections, and parameter references.
- Persists across examples
- Defines possible data flow
- Can include conditional branches
- May be traced or built eagerly
Forward values
Concrete tensors produced during one execution.
- Depend on the current input
- May depend on random masks
- Can be cached for gradients
- Often dominate activation memory
Example
A forward pass may change between training and evaluation
Identical inputs do not always imply identical internal values.
- Dropout is not merely “usually” disabled at evaluation; it is defined not to run. PyTorch's reference for torch.nn.Dropout states: “Furthermore, the outputs are scaled by a factor of 1/(1-p) during training. This means that during evaluation the module simply computes an identity function.” Keras 3 documents the same contract independently: kept units are scaled up by 1 / (1 - rate) “such that the sum over all inputs is unchanged”, and “Note that the Dropout layer only applies when training is set to True in call(), such that no values are dropped during inference.” Two frameworks, written separately, one contract.
- Batch normalization runs a different function in the two modes, and its authors gave the reason. Ioffe and Szegedy, in 2015: “The normalization of activations that depends on the mini-batch allows efficient training, but is neither necessary nor desirable during inference; we want the output to depend only on the input, deterministically.” PyTorch's BatchNorm2d carries that out. During training the layer “keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation”, with a default momentum of 0.1. The same paper reports that the technique “achieves the same accuracy with 14 times fewer training steps”, and an ensemble “reaching 4.82% top-5 test error”.
- Random data augmentation changes the input before it reaches the model.
- Sampling-based generative heads can choose different outputs from the same logits.
- Explicit random seeds improve reproducibility but do not replace train/eval mode control. On their own they are not enough. PyTorch's reproducibility notes: “Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds.” TensorFlow says the same in tf.config.experimental.enable_op_determinism: “By default, op determinism is not enabled, so ops might return different results when run with the same inputs. These differences are often caused by the use of asynchronous threads within the op nondeterministically changing the order in which floating-point numbers are added.” Determinism is off by default in both frameworks, and switching it on costs performance.
Why forward values consume training memory
Reverse-mode differentiation needs selected forward inputs or outputs to compute local derivatives later. So frameworks retain those values until the backward pass finishes. PyTorch states the default without hedging: “By default, tensors computed during the forward pass are kept alive until they are used in gradient computations in the backward pass.” JAX describes the same behaviour from its own side: “When differentiating a function in reverse-mode, by default all the linearization points (e.g. inputs to elementwise nonlinear primitive operations) are stored when evaluating the forward pass so that they can be reused on the backward pass.”
Gradient checkpointing saves fewer intermediate tensors and recomputes the rest. It changes resource use, not the mathematical function. The PyTorch page for torch.utils.checkpoint puts it in one line: “Activation checkpointing is a technique that trades compute for memory.” The trade was priced in 2016: “we can reduce the memory cost of a 1,000-layer deep residual network from 48G to 7G with only 30 percent additional running time cost on ImageNet problems”. Tianqi Chen and three co-authors were not offering a rule of thumb — “we design an algorithm that costs O(sqrt(n)) memory to train a n layer network, with only the computational cost of an extra forward pass per mini-batch”.
The same decision has been priced at the other end of the scale. In 2023 Korthikanti and six co-authors combined sequence parallelism with selective recomputation. That combination “reduces activation memory by 5x, while reducing execution time overhead from activation recomputation by over 90%”. Their test was a 530-billion-parameter GPT-3-style model. It was trained across 2,240 NVIDIA A100 GPUs. Model flops utilisation went from 42.1% under ordinary full recomputation to 54.2%. That is “29% faster”, from a decision about which intermediate tensors to keep.
Figure
Training memory includes parameters, gradients, optimizer state, and retained activations.
Analogy
A recipe with forks and saved ingredients
A recipe divides one sauce into two pans, transforms each portion, and combines them later. Notes beside each pan record what was used, so a mistake in the finished sauce can be traced backward to the step that produced it.
A recipe also misses automatic derivative accumulation and parameter reuse. Kitchens do not calculate how the final dish changes with every ingredient quantity. And no kitchen has ever had to decide, as Tianqi Chen and his co-authors did, whether to keep every pan on the counter or cook a few steps again to free the space.
Branches and merges are ordinary program structure, not mysterious neural behavior.
Key idea
Instrument the forward pass before guessing
Add hooks or logging at module boundaries to capture shapes, ranges, finite-value checks, and selected activation summaries. Begin with one deterministic batch. A seed alone does not give you one: op determinism is off by default, and CPU and GPU results may differ under identical seeds.
Dumping every tensor can overwhelm memory and bury the signal. The reason is the one already measured: storage grows with the number of operations executed. Instrument around the suspected boundary, then narrow the trace.
A small, intentional trace is more useful than an uncontrolled activation archive.
Key takeaways
- A forward pass is an ordered execution of tensor operations defined by data dependencies. In He's residual block that order ends at y = F(x, {Wi}) + x — a shortcut connection plus element-wise addition.
- Computational graphs contain chains, branches, merges and reused values. The Inception module branches into 1×1, 3×3 and 5×5 filters and a pooling path, then merges by concatenation into a single output vector.
- The persistent graph is not the temporary activations produced for one input. TensorFlow's graph is a data structure of tf.Operation and tf.Tensor objects that outlives any batch; the values are made again on every call.
- Training and evaluation execute different behavior by documented contract: dropout scales surviving activations by 1/(1-p) in training and computes the identity at evaluation, and batch normalization switches to running estimates of mean and variance at inference.
- Backward computation requires retained forward values, and the cost is structural: storage grows in the worst case in proportion to the number of operations evaluated, while arithmetic goes up by no more than a small constant factor.
- Targeted instrumentation gives stronger debugging evidence than treating a model call as a black box, and a controlled batch takes more than a seed.