Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution

AI summary

Overview

Research area: On-device machine learning systems — specifically, latency optimization for deep neural network inference on mobile platforms through heterogeneous CPU-GPU execution.

Technical level: Advanced. The paper assumes familiarity with GPU execution models (workgroups, kernels, dispatch), OpenCL shared virtual memory, gradient-boosted decision trees, and quantization-free neural network layer computation.

Scope: The paper proposes a system that splits individual linear and convolutional layers across a mobile CPU and GPU, using white-box latency predictors derived from TFLite kernel source code and a low-overhead synchronization mechanism built on OpenCL fine-grained shared virtual memory.

Publication venue (as reported): Selected Papers of EPEW 2025, Lecture Notes in Computer Science, volume 15657, pages 41–55, Springer, 2026. arXiv identifier: arXiv:2510.21081v2 [cs.LG], dated 18 Feb 2026.

What This Paper Is About

Mobile devices have limited compute resources, but their unified memory architecture lets the CPU and GPU share main memory directly, and mobile CPUs using XNNPACK in TensorFlow Lite can be competitive with mobile GPUs for some operations. This creates an opportunity to run parts of a neural network layer on the CPU and the rest on the GPU at the same time to lower inference latency.

Two obstacles block this: predicting how long a GPU kernel will take is hard because GPU latency curves contain sudden spikes caused by heuristic workgroup selection and kernel switching, and the synchronization needed to combine partial CPU and GPU results is expensive. The paper tackles both, aiming to choose good CPU-GPU splits quickly and cheaply.

Key Contributions

1. Accurate latency predictors using black-box/white-box kernel detail. The authors developed predictors that use kernel implementation details and kernel dispatch behavior from TFLite, and train separate predictors for each kernel implementation. These capture discontinuities caused by heuristic workgroup choices and kernel selection.

2. A lightweight CPU-GPU synchronization mechanism. Using OpenCL fine-grained shared virtual memory, the design avoids expensive data mapping operations for cache coherence and avoids notification delay through active polling. On a Motorola Edge Plus 2022 smartphone, synchronization overhead for linear operations is reduced from 162 μs to 7 μs.

3. A comprehensive measurement dataset. Latency measurements were collected for 2,039 linear and 2,051 convolution operations across four mobile devices (Pixel 4, Pixel 5, Motorola Edge Plus 2022, OnePlus 11), using co-execution strategies with 1 to 3 CPU threads plus the GPU.

4. A comprehensive evaluation. The predictors quickly select co-execution strategies achieving up to 1.89x speedup for linear operations and 1.75x for convolution operations on Pixel 5 smartphones, compared to maximum values of 2.01x and 1.87x found by exhaustive grid search.

Main Findings

• CPU-GPU performance gap is narrow on mobile. For matrix multiplications of sizes 50 × 3072 and 3072 × C_out, a CPU implementation with 3 threads achieves lower latency than a GPU kernel when C_out < 425 (measured on OnePlus 11, with 95% confidence intervals).

• GPU latency is highly non-monotonic. In one experiment, a linear operation with C_out = 2500 was 1.85 times slower than one with C_out = 2520, despite being smaller.

• Naive ML predictors make poor partitions. A gradient-boosted decision tree using matrix sizes as features only captured the overall increasing latency trend. It assigned 2,378 output channels to the GPU (measured 483 μs) and 694 to the CPU (measured 424 μs), achieving only 1.02x speedup over GPU-only execution.

• Feature augmentation fixes partitioning. With the augmented predictors, 2,480 channels were assigned to the GPU (measured 379 μs; prediction improved from 481 μs to 375 μs) and 592 to the CPU (measured 354 μs), improving speedup from 1.02x to 1.29x (495 μs to 393 μs).

• Two causes of GPU latency discontinuity were identified from TFLite source code: heuristic workgroup choices (a strong correlation between number of workgroups and kernel latency was observed) and kernel selection (for a 3×3 filter with input size 64×64×128, TFLite switches to the Winograd algorithm when output channels exceed 128).

• Synchronization overhead dominates without the proposed mechanism. Passive waiting via the OpenCL clWaitForEvents API incurred average overhead of 162 μs across 2,039 linear layers and 141 μs across 2,051 convolutional layers, accounting for 39.9% and 15.8% of total co-execution latency respectively with 1 CPU thread. Active polling reduced this to averages of 7.0 μs (linear) and 5.4 μs (convolutional).

• Prediction accuracy (MAPE) varies by device and layer type. On linear layers: Pixel 4 GPU 4.4%, 1 CPU 11.5%, 2 CPUs 7.1%, 3 CPUs 5.8%; Pixel 5 3.7%, 6.2%, 7.8%, 7.2%; Moto 2022 4.0%, 2.5%, 2.6%, 2.4%; OnePlus 11 3.7%, 3.1%, 2.9%, 3.1%. On convolutional layers errors were generally higher: Pixel 4 8.5%, 11.4%, 8.8%, 7.2%; Pixel 5 7.7%, 6.9%, 8.1%, 7.1%; Moto 2022 9.0%, 4.0%, 3.6%, 3.5%; OnePlus 11 7.4%, 4.8%, 4.2%, 4.4%.

• Per-layer co-execution speedups. Pixel 4: GBDT 1.21x/1.52x/1.84x (1/2/3 threads) for linear and 1.22x/1.46x/1.69x for convolutional, versus grid search 1.29x/1.59x/1.92x and 1.31x/1.56x/1.79x. Pixel 5: GBDT 1.51x/1.78x/1.89x and 1.45x/1.69x/1.75x, versus search 1.63x/1.92x/2.01x and 1.49x/1.80x/1.87x. Moto 2022: GBDT 1.20x/1.32x/1.44x and 1.16x/1.27x/1.39x, versus search 1.23x/1.36x/1.49x and 1.22x/1.34x/1.46x. OnePlus 11: GBDT 1.06x/1.17x/1.26x and 1.07x/1.22x/1.35x, versus search 1.13x/1.25x/1.35x and 1.12x/1.27x/1.40x.

• Speedups are higher on devices with a smaller CPU/GPU performance gap (Pixel 4 and Pixel 5), because a larger fraction of the task can be offloaded to the CPU.

• End-to-end model speedups with GPU plus 3 CPU threads. From baseline to co-execution: Pixel 4 VGG16 83.3 ms → 73.0 ms (1.14x end-to-end), ResNet-18 17.5 ms → 11.4 ms (1.54x), ResNet-34 37.5 ms → 22.5 ms (1.67x), Inception-v3 99.5 ms → 61.5 ms (1.62x). Pixel 5 VGG16 194.8 ms → 125.1 ms (1.56x), ResNet-18 33.2 ms → 18.6 ms (1.78x), ResNet-34 65.2 ms → 37.1 ms (1.76x), Inception-v3 184.3 ms → 102.9 ms (1.79x). Moto 2022 VGG16 32.0 ms → 29.7 ms (1.08x), ResNet-18 7.5 ms → 6.7 ms (1.11x), ResNet-34 14.7 ms → 12.9 ms (1.14x), Inception-v3 52.0 ms → 41.1 ms (1.27x). OnePlus 11 VGG16 27.4 ms → 26.2 ms (1.05x), ResNet-18 8.5 ms → 6.8 ms (1.25x), ResNet-34 17.6 ms → 13.8 ms (1.27x), Inception-v3 44.2 ms → 37.8 ms (1.17x).

• End-to-end gains are slightly lower than per-operation gains, attributed to memory access overhead between layers. Pooling operations are always scheduled on the GPU because their latency is negligible and this avoids synchronization overhead.

• Feature augmentation reduces prediction error. In the ablation study on Moto 2022, augmentation reduced MAPE of linear layers from 9.3% to 4.4% and of convolutional layers from 14.1% to 9.3%. For convolutional layers using the GPU and 1 CPU thread, latency reduction improved from 1.08x to 1.16x.

• Baseline synchronization destroys gains. Without the overhead reduction, speedups on Moto 2022 were 0.76x/0.81x/0.88x for linear and 0.98x/1.07x/1.17x for convolutional — below 1x in most linear cases.

• A comparison to prior work on VGG16: related work using the MACE ML framework reduced Pixel 4 VGG16 latency from a baseline of 200 ms to around 150 ms. In this work, the Pixel 4 VGG16 baseline was 83.3 ms, reduced to 73.0 ms. The difference in baseline is attributed to TFLite's use of image storage types to exploit L1 texture cache and its efficient Winograd kernels.

• Partitioning decisions are fast enough for offline use: the GBDT predictors typically take 3–4 ms to determine the optimal partitioning for each operation, and these decisions can be made offline before deployment as part of the compilation process.

Methodology in Plain English

The authors split a layer's work by output channels. Each output channel corresponds to a distinct column of a linear weight matrix or a distinct convolution kernel, so the first C_CPU channels go to the CPU and the remaining C_GPU channels go to the GPU, with the total satisfying C_CPU + C_GPU = C_out. Both devices read the same shared input. The goal is to pick the split that minimizes the overhead plus the maximum of the CPU and GPU times, since the parallel section finishes when the slower device finishes.

To predict GPU latency, the team read TFLite's source code and identified how the framework chooses among three convolution kernel implementations (conv_constant, winograd, conv_generic) and how it picks workgroup sizes heuristically based on hardware factors such as registers, compute unit occupancy, and memory access patterns. They then built separate predictors per kernel implementation and added dispatch-related features (workgroup size and number of workgroups) alongside operation dimensions. Predictors are gradient-boosted decision trees trained with LightGBM and tuned with Optuna.

For synchronization, they store layer outputs in OpenCL fine-grained shared virtual memory so the CPU and GPU read and write the same main memory region without copies and without explicit mapping and unmapping, since hardware guarantees cache coherence. To eliminate notification delay, they dispatch a polling kernel after each GPU computation that updates two flags (cpu_flag and gpu_flag) in fine-grained SVM, requiring busy waiting on both CPU and GPU. This costs extra power when workloads are unbalanced, which the accurate latency predictions help mitigate.

Data collection used a C++ benchmarking tool built on TFLite, co-executing OpenCL kernels from the TFLite GPU Delegate with CPU kernels from XNNPACK. Measurements were stabilized by enabling performance mode on the GPU and all CPU cores, pinning CPU threads to high-performance cores, attaching an external cooling fan, and allowing a 1-second cool-down after profiling each operation configuration. Training data was generated by structured random sampling from power-of-two intervals, yielding 12,500 configurations per layer type with 20% held out for testing. Grid search baselines used a step size of 8 over [0, C_out] and were evaluated on a random 10% subset of test cases.

Why This Matters

Impact on research: The paper shows that treating GPU latency prediction as a black box is insufficient for heterogeneous scheduling — the discontinuities that matter come from framework implementation choices, which can be read out of source code. It also demonstrates that synchronization overhead, not just compute time, is the binding constraint for fine-grained co-execution on mobile.

Real-world applications:

• On-device inference for vision models such as VGG16, ResNet-18, ResNet-34, and Inception-v3 on smartphones.
• Privacy-preserving and offline-capable mobile applications, which the authors list as motivations for on-device deployment.
• Real-time applications requiring low latency, also cited by the authors as a benefit of on-device deployment.
• Deployment of large models such as LLMs and LVMs on mobile platforms, which the authors identify as a growing demand.

Industry relevance: The technique builds directly on TFLite's GPU Delegate and XNNPACK and produces partitioning decisions that can be computed offline during the compilation process, which fits existing mobile deployment pipelines. The measured platforms (Pixel 4, Pixel 5, Motorola Edge Plus 2022, OnePlus 11) span multiple Android vendors, and the ablation shows gains are largest on devices where CPU and GPU are comparably powerful.

Future Directions

• Parallel execution across CPU, GPU, and NPU, which the authors state as future work.
• Investigating the effects of model quantization, also stated as future work.
• Reducing the end-to-end gap between per-operation and full-model speedups, which the authors attribute to memory access overhead between layers.
• Reducing the power cost of busy-wait polling under unbalanced workload partitioning, an issue the authors acknowledge and currently address only indirectly through better latency prediction.

Target Audience

Researchers and engineers working on mobile and edge inference systems, heterogeneous CPU-GPU scheduling, and on-device ML deployment. It is also relevant to practitioners using TFLite's GPU Delegate who want to understand when CPU-GPU co-execution pays off, and to systems researchers interested in latency modeling of GPU kernels or low-overhead inter-processor synchronization on unified memory architectures. Readers need background in GPU execution models and ML inference runtimes to follow the kernel characterization and partitioning formulation.

Authors’ abstract

Deploying deep neural networks on mobile devices is increasingly important but remains challenging due to limited computing resources. On the other hand, their unified memory architecture and narrower gap between CPU and GPU performance provide an opportunity to reduce inference latency by assigning tasks to both CPU and GPU. The main obstacles for such collaborative execution are the significant synchronization overhead required to combine partial results, and the difficulty of predicting execution times of tasks assigned to CPU and GPU (due to the dynamic selection of implementations and parallelism level). To overcome these obstacles, we propose both a lightweight synchronization mechanism based on OpenCL fine-grained shared virtual memory (SVM) and machine learning models to accurately predict execution times. Notably, these models capture the performance characteristics of GPU kernels and account for their dispatch times. A comprehensive evaluation on four mobile platforms shows that our approach can quickly select CPU-GPU co-execution strategies achieving up to 1.89x speedup for linear layers and 1.75x speedup for convolutional layers (close to the achievable maximum values of 2.01x and 1.87x, respectively, found by exhaustive grid search on a Pixel~5 smartphone).