Deep architectures
Bottlenecks, Depthwise Convolution, and Efficient CNN Blocks
Compare bottleneck, grouped, depthwise-separable, inverted-residual, and compound-scaling designs through accuracy, memory traffic, and hardware efficiency.
By the end you can
- Decompose standard and depthwise-separable convolution into channel and spatial mixing
- Explain bottleneck and inverted-residual block designs
- Distinguish theoretical operation counts from measured hardware latency
- Choose efficient blocks using target-device profiling rather than FLOPs alone
Comparison
How convolutional blocks factor the work
Efficiency designs decide where spatial mixing and channel mixing happen. Each of the four arrangements below arrived in a named paper, for a stated reason.
Grouped convolution was not invented to save arithmetic. It was invented to fit a network onto two graphics cards. AlexNet was split across two NVIDIA GTX 580 GPUs in 2012, and the paper gives the reason: “A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the networks that can be trained on it.” The scheme that squeezed the model into that budget is the ancestor of every grouped convolution since. Kernels in layer 4 read only from kernel maps residing on the same GPU.
The representational isolation listed on the second card came later, and as an observation. The two groups specialised on their own. One set of first-layer kernels came out largely colour-agnostic, the other largely colour-specific. The network won ILSVRC-2012 with a 15.3% top-5 test error against 26.2% for the second-best entry. The challenge's own results table records SuperVision at 0.15315 and the runner-up ISI at 0.26172, with full results released on 13 October 2012. A memory workaround won the competition.
The bottleneck block has an equally specific definition, and an equally specific price. He and three colleagues set it out in 2016: “The three layers are 1×1, 3×3, and 1×1 convolutions, where the 1×1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3×3 layer a bottleneck with smaller input/output dimensions.”
What that compression bought is on the record. ResNet-50 runs at 3.8 billion FLOPs, ResNet-152 at 11.3 billion. Both stay below VGG-16 and VGG-19, at 15.3 and 19.6 billion. The ensemble took 3.57% top-5 error and first place in ILSVRC 2015 classification, recorded as 0.03567 for MSRA in the challenge's results table.
The residual path in the fourth card is not decoration either. The same authors note that replacing the identity shortcut with a projection at the two high-dimensional ends would double time complexity and model size. The narrow interface is what makes the shortcut affordable.
Standard convolution
Mixes space and channels in one dense operator.
- Strong fused implementation support
- Parameter and arithmetic cost grow with both channel dimensions
- Simple tensor interface
- Often competitive on accelerators
Grouped convolution
Partitions channels into smaller independent groups.
- Reduces dense channel interaction
- Can improve parallel structure
- Requires later cross-group mixing
- Grouping choice affects representational isolation
Depthwise separable convolution
Applies one spatial filter per channel, then uses pointwise mixing.
- Large arithmetic reduction in common settings
- Separates spatial and channel operations
- May become memory-bound
- Kernel support determines real speed
Bottleneck block
Projects to a narrower or wider hidden space around an expensive operator.
- Controls internal compute
- Creates compression or expansion interfaces
- Can restrict information if too narrow
- Often paired with residual paths
Why mobile blocks often expand before filtering
An inverted residual block usually expands a compact input into a wider hidden representation. It then performs inexpensive depthwise spatial mixing and projects back to a narrow output. The residual path connects the compact endpoints when shapes agree.
This reverses the classic bottleneck intuition. The nonlinear transformation receives more channels, while the externally stored representation stays compact.
Internal expansion can increase representational capacity without keeping every stage wide.
Case
MobileNetV2 moved the shortcuts onto the thin layers and stripped their nonlinearities
The pattern got its name in 2018: MobileNetV2. The architecture is “based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers”, and its “intermediate expansion layer uses lightweight depthwise convolutions to filter features”. Sandler and four co-authors add a requirement that is easy to miss: “it is important to remove non-linearities in the narrow layers in order to maintain representational power”. That is the second half of the paper's title — inverted residuals and linear bottlenecks.
Those are the camera-ready words, and it is worth knowing which words you are quoting. The preprint went up on 13 January 2018 with the same three facts in different wording. There, the input and output of the residual block are thin bottleneck layers, and the lightweight depthwise convolutions filter features in the intermediate expansion layer. Only the sentence about removing non-linearities in the narrow layers is word-for-word identical on both hosts.
Visual
From paper efficiency to device efficiency
Each step below can overturn the ranking that operation counts suggest. One team wrote that overturning into the search itself, rather than into a discussion section. MnasNet executed candidate architectures on Pixel phones and timed them inside the objective function. The 2019 abstract opens with the decision: “Unlike previous work, where latency is considered via another, often inaccurate proxy (e.g., FLOPS), our approach directly measures real-world inference latency by executing the model on mobile phones.” The result they report is 75.2% ImageNet top-1 accuracy at 78ms latency on a Pixel phone. That is 1.8× faster than MobileNetV2 with 0.5% higher accuracy, and 2.3× faster than NASNet with 1.2% higher accuracy.
The last box of the diagram is also where the numbers get decided, not merely reported. Google's own announcement of the work, a year earlier on 7 August 2018, quoted the pre-squeeze-and-excitation version of the same result at 1.5× faster than MobileNetV2 and 2.4× faster than NASNet. The ratios moved with the version of the model. What did not move is that both figures came off a stopwatch attached to a named phone. Neither could have been derived from an operation count.
- 1
Analytical cost
Count parameters, multiply–accumulates, and activation sizes.
- 2
Kernel implementation
Check whether operators are fused and optimized for the target runtime.
- 3
Memory movement
Measure reads, writes, layout changes, and intermediate materialization.
- 4
Execution context
Profile batch size, precision, compiler, thermal state, and concurrency.
- 5
End-to-end latency
Include preprocessing, transfers, postprocessing, and scheduling.
Example
Why “lightweight” blocks behave differently in practice
A design earns the label efficient only inside a declared deployment environment. One published experiment timed the same models in two of them. More than 40 ImageNet-1k architectures ran on a workstation with an NVIDIA Titan X Pascal — which the authors call Titan Xp — and on an NVIDIA Jetson TX1 embedded board, at batch sizes 1 through 64. Bianco and three colleagues published the outcome in 2018, in their list of key findings: “almost all models are capable of real-time or super real-time performance on a high-end GPU, while just a few of them can guarantee them on an embedded system”. The same list records that recognition accuracy does not increase as the number of operations increases. SENet-154 needs about 3× the operations of SE-ResNeXt-101(32x4d) for almost the same accuracy.
- High-end GPU: at batch size 1, every model in that study except SENet-154 clears the paper's super real-time bar of more than 60 FPS on the Titan Xp. The workstation flattens almost every difference between the architectures.
- Embedded board: on the Jetson TX1 the same bar is cleared only by the SqueezeNets, the MobileNets, ResNet-18, GoogLeNet and AlexNet. Same weights, same models, inverted verdict.
- Microcontroller: the MCUNet paper states the budget in 2020 — “a state-of-the-art ARM Cortex-M7 MCU only has 320kB SRAM and 1MB Flash storage”. STMicroelectronics' own datasheet for the STM32F746 confirms it: up to 1 Mbyte of flash and 320 Kbytes of SRAM.
- Two different failures on one chip: on that MCU, ResNet-50 exceeds the storage limit by 100×. MobileNetV2 — the inverted-residual, linear-bottleneck design this lesson is built around — exceeds the peak memory limit by 22×, and its int8-quantised version still exceeds it by 5.3×.
- Batch regime: some Jetson TX1 entries are missing from the benchmark tables outright, because there was not enough system memory to process the larger batches. The deployment answer arrives as an absence rather than as a slow number.
Analogy
A workshop that separates cutting from assembly
One station in a workshop shapes each component and another assembles the relationships between components. Specialized stations can reduce duplicated work.
Hardware may spend more time moving tensors between stations than computing. Factorization helps only when the code that runs it preserves the expected efficiency.
An efficient mathematical factorization can become an inefficient memory schedule.
Key idea
FLOPs are not latency
FLOP counts ignore memory bandwidth, kernel launch overhead, parallel occupancy, sparsity support, and compiler fusion. They also omit non-convolutional work around the model.
Report measured latency and energy on the actual device, with realistic input sizes and batch behavior. Analytical counts remain useful for diagnosis, not as a substitute for profiling.
ShuffleNet V2 ran the controlled experiments for this in 2018. Ma and three colleagues begin from a diagnosis: “network architecture design is mostly guided by the indirect metric of computation complexity, i.e., FLOPs”, while “the direct metric, such as speed, also depends on the other factors such as memory access cost and platform characterics” (their spelling). They also say why the indirect one misses. “the FLOPs metric only account for the convolution part”, whereas “data I/O, data shuffle and element-wise operations (AddTensor, ReLU, etc) also occupy considerable amount of time”.
Two of their measurements make it concrete, and neither changes the convolution FLOPs. On GPU, a “4-fragment structure is 3× slower than 1-fragment”. On a ResNet bottleneck unit, “around 20% speedup is obtained on both GPU and ARM, after ReLU and shortcut are removed”. Both times, the thing that got faster is the part the count does not count.
The target hardware is part of the architecture specification.
Figure
Steps
Run an architecture benchmark that survives contact with production
A reproducible benchmark controls the conditions that make efficiency claims meaningful. An industry has already written its version of these five steps down. MLPerf Inference — driven by more than 30 organisations and more than 200 ML engineers and practitioners — defines four scenarios: single-stream, multistream, server and offline. Only single-stream's metric is itself a tail-latency percentile, and its 2020 definition is one line: “The metric is the query stream's 90th-percentile latency.” It requires 1,024 queries. Multistream reports the number of streams sustainable under a latency bound. Server reports queries per second under a latency bound. Offline reports throughput with latency unconstrained — 1 query with at least 24,576 samples.
Step 3 above therefore has a price list. The latency bounds are enforced at tail percentiles rather than means. The number of queries scales with how far into the tail you want to see: 1,024 for a 90th percentile, and 262,742 for a 99th-percentile guarantee, rounded up to 270K. A median from a handful of warm runs is not a cheaper version of this measurement. It is a different one.
And the rules were written to be executed by other people, which is the part that makes them a standard. The first call for submissions returned more than 600 reproducible inference-performance measurements, from 14 organizations across over 30 systems. The consortium behind the benchmark — MLPerf then, MLCommons today — published 595 validated results on 6 November 2019.
1. Freeze the tensor contract
Use the actual resolution, sequence length, precision, and batch regime.
2. Warm the runtime
Separate compilation and initialization from steady-state execution.
3. Measure distributions
Record median and tail latency, not one favorable run.
4. Track memory and energy
Capture peak activations, transfers, and sustained device behavior.
5. Compare equal-quality points
Evaluate models at comparable task performance rather than equal parameter count alone.
Position
No architecture is efficient; only an architecture on a device is
The vocabulary gives the mistake away. Lightweight, mobile, efficient — the adjectives attach to the block, as if efficiency travelled with the design from the paper to the phone. The papers behind this lesson do not support that reading. One of them withdraws it from the very metric the field was optimising.
That paper is ShuffleNet V2, in 2018. Ma and three colleagues named the problem: “network architecture design is mostly guided by the indirect metric of computation complexity, i.e., FLOPs”, while “the direct metric, such as speed, also depends on the other factors such as memory access cost and platform characterics”. Their decomposition says why. The count covers only the convolution part. Data input and output, data shuffle and elementwise operations take a considerable share of the clock. Two of their measurements are worth carrying around. A four-fragment block ran three times slower than a one-fragment block on GPU. Removing the ReLU and the shortcut from a ResNet bottleneck bought roughly 20 per cent on both GPU and ARM. That is a change to two elementwise operations, which is exactly the part the FLOP count does not count.
The MobileNetV2 block makes a second point structurally, without measuring anything. Its shortcuts run between the thin bottleneck layers while the intermediate expansion layer does the filtering. What is carried between blocks and what is computed inside one are deliberately different sizes. A design built that way cannot be summarised by a single notion of small. Parameters, peak activation memory and latency are three quantities. They move independently, and a spec sheet usually reports the first.
MCUNet measured how far apart the three can travel. On one ARM Cortex-M7 microcontroller with 320kB SRAM and 1MB Flash, ResNet-50 exceeds the storage limit by 100× while MobileNetV2 exceeds the peak memory limit by 22×. Two models, two different axes, one chip. The Titan Xp and Jetson study found the same independence in the other direction. At batch size 1, all but one of more than 40 architectures cleared the super real-time bar on the workstation GPU. On the Jetson TX1, five families did.
Which is why the benchmark above is not a formality bolted on at the end. It is the step that gives the word efficient a referent, and it can be run to somebody else's rules. MLPerf's single-stream scenario asks for 1,024 queries and reports the 90th percentile, and the first round of submissions produced more than 600 measurements from 14 organizations. The MnasNet team went further and moved the measurement upstream, timing candidates on Pixel phones inside the search and reporting 78ms. Until a latency and an energy figure exist for the device, the batch size and the runtime that will actually ship, an efficiency claim is a claim about arithmetic that no particular machine has agreed to perform.
Ask which device and which batch size, or the word efficient has nothing to point at.
Depth, width, and resolution should be scaled together deliberately
Increasing only depth changes optimization and receptive fields. Increasing only width changes channel capacity and activation cost. Increasing resolution changes the amount of evidence and the quadratic size of some later operations.
Compound scaling is a disciplined search principle. It is not a law that one coefficient fits every dataset or device. Re-profile after any change to input resolution or precision.
Compound scaling was formalised in 2019. Tan and Le propose “a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient”, which is to say it does not enlarge one dimension at a time. The family it produced is EfficientNet. Of its largest member they report that EfficientNet-B7 “achieves state-of-the-art top-1 / top-5 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet”.
Note what the coefficient is. It was searched on their baseline, not derived from a law. Note also which figures survive a change of version. The headline top-1 accuracy reads 84.4% in the conference text and 84.3% in the latest preprint revision, while the 8.4× and 6.1× ratios are identical in both. When a number moves between two printings of the same paper, it will not hold still across your dataset either.
Scaling dimensions interact; an architecture family is a curve of quality–cost points, not a single model.
Key takeaways
- Efficient CNN blocks factor spatial and channel mixing to alter arithmetic and representation flow. Grouped convolution began as a way to fit AlexNet across two GTX 580 cards holding 3GB each.
- Inverted residual designs keep external tensors compact while using a wider nonlinear hidden space. MobileNetV2 removes the non-linearities from the narrow layers to maintain representational power.
- FLOPs and parameter counts do not capture memory movement, kernel quality, or runtime overhead. ShuffleNet V2's 4-fragment structure ran 3× slower than 1-fragment on GPU, with the convolution FLOPs unchanged.
- Latency claims require the real device, compiler, precision, input size, and batch regime. MnasNet reported 75.2% top-1 at 78ms by executing candidates on Pixel phones rather than trusting a proxy.
- Peak activation memory can disqualify a model whose stored parameter file looks small. On a 320kB-SRAM Cortex-M7, MobileNetV2 exceeds the peak memory limit by 22×, and by 5.3× even after int8 quantisation.
- Depth, width, and resolution interact, so scaling should be profiled as one coordinated decision and reported the way MLPerf demands — at a tail percentile, over 1,024 queries for the 90th and 270K for the 99th.