Edge Deployment, Compression, and Monitoring

A deployment budget should trace every stage on target hardware.

The model is one of five stages, and on a constrained device it is rarely the only one that misses the budget. A stage that never appears in the architecture diagram — a video decode, a colour conversion, a memory copy, a serialization before the network — is charged to the same milliseconds as the kernels.

The best method depends on model, hardware, calibration data, and failure tolerance.

Integer-only inference has a published recipe, and half of it gets forgotten. Jacob and co-authors proposed in 2018 “a quantization scheme that allows inference to be carried out using integer-only arithmetic, which can be implemented more efficiently than floating point inference on commonly available integer-only hardware”. The other half is the part that gets dropped when people quote the speedup: they “co-design a training procedure to preserve end-to-end model accuracy post quantization”. Training is in the recipe.

The numbers are documented, and so are the caveats. Google's LiteRT documentation quotes “4x smaller, 3x+ speedup” for full integer quantization and “4x smaller, 2x-3x speedup” for dynamic range quantization. ONNX Runtime warns that “Quantization is not a loss-less transformation. It may negatively affect a model’s accuracy.” and that “Older hardware will not benefit from quantization.” The gains are tied to specific instructions: x86-64 with VNNI, GPUs with Tensor Core int8 support, Arm dot-product instructions.

Hardware support is what turns sparsity into speed, and only one pattern of it. Mishra and co-authors describe “Sparse Tensor Cores, which exploit a 2:4 (50%) sparsity pattern that leads to twice the math throughput of dense matrix units”, together with “a simple workflow for training networks that both satisfy 2:4 sparsity pattern requirements and maintain accuracy”. NVIDIA states the requirement plainly: “In each contiguous block of four values, two values must be zero”, with the Sparse Tensor Cores “operating only on the nonzero values in the compressed matrix”. Remove half the weights in any other arrangement and the Ampere hardware has nothing to exploit. The FLOPs fall and the latency does not.

On-device processing can reduce raw-data transfer, latency, and how much the system depends on connectivity; cloud services can provide larger models, shared updates, centralized monitoring, and cross-device context.

The best-documented case of a partition that broke on its cloud leg is a screening deployment, not a thought experiment. A Google deep-learning screener for diabetic retinopathy was studied in use across eleven clinics in Thailand, and the study reported a collision between a model's specification and a building: “We find tensions between the model’s thresholds for data quality, and the quality of data that arise from an imperfect, resource-constrained environment.” MIT Technology Review, reporting on the study in 2020, recorded both halves of the gap. The lab system reached “more than 90% accuracy”. In the field “more than a fifth of the images were rejected” — and “Because the system had to upload images to the cloud for processing, poor internet connections in several clinics also caused delays”. Neither the rejection threshold nor the upload was a modelling error. Both were partition decisions taken before anyone saw the clinics.

Partition by privacy, bandwidth, power, failure mode, update cadence, and evidence needs; define behavior when connectivity is slow, absent, or untrusted.

Average accuracy can hide capability loss, and the hiding has been measured and named. In 2019 Hooker and colleagues compared pruned and quantized networks against their uncompressed originals and reported the shape of the damage: “We find that models with radically different numbers of weights have comparable top-line performance metrics but diverge considerably in behavior on a narrow subset of the dataset.” They named that subset Pruning Identified Exemplars — PIEs — and found it over-indexes on the underrepresented long tail. The top-line metric is not wrong. It is averaging over a population in which the loss is concentrated.

An independent group reproduced the non-uniformity on class recall. Writing in 2022, they note that “the impact of network pruning is not uniform: prior work has shown that the recall for underrepresented classes in a dataset may be more negatively affected”. They report that pruning worsens recall for classes whose recall already sits below overall accuracy, and improves it for classes above it. Compression does not degrade a model evenly. It sharpens whatever imbalance the model already had. That is why the slices below have to be evaluated separately rather than trusted to the mean.

Treat conversion and compression as new model versions. This is not a house style. A regulator, a vendor's own post-mortem and three medicines agencies have each written the same thing down.

A software update can be a safety recall. On 12 December 2023 Tesla filed Part 573 Safety Recall Report 23V-838 with NHTSA, covering 2,031,220 vehicles — Model S, Model X, Model 3 and Model Y from model years 2012 to 2023, all equipped with Autosteer. The filing followed a NHTSA preliminary evaluation opened on 13 August 2021 and upgraded to an engineering analysis on 8 June 2022. The recalled component is listed simply as “Vehicle Software”. The entire remedy is a version string: “At no cost to customers, affected vehicles will receive an over-the-air software remedy, which is expected to begin deploying to certain affected vehicles on or shortly after December 12, 2023, with software version 2023.44.30.” Two million devices, and the unit of the safety action is a build number.

Step 5 costs the most when it is skipped. At 04:09 UTC on Friday 19 July 2024 CrowdStrike released a Rapid Response Content configuration update — Channel File 291 — that crashed Windows hosts running sensor 7.11 and above which were online. The defect was reverted at 05:27 UTC. Microsoft's David Weston wrote on the company blog the next day: “We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices, or less than one percent of all Windows machines.” The first item on CrowdStrike's own published remediation list is the step it had not taken: “Implement a staggered deployment strategy for Rapid Response Content in which updates are gradually deployed to larger portions of the sensor base, starting with a canary deployment.”

In medical devices the plan is a filing. The U.S. Food and Drug Administration asks a manufacturer to “describe the planned AI-DSF modifications, the associated methodology to develop, validate, and implement those modifications, and an assessment of the impact of those modifications” in a predetermined change control plan, a PCCP. The guidance was issued on 4 December 2024 and reissued on 18 August 2025. One sentence in it makes retraining a regulatory event rather than a deployment detail: “For purposes of this guidance, a PCCP includes those device modifications that generally would otherwise require a new marketing submission.”

Three regulators converged on the same checklist independently. On 24 October 2023 the FDA, Health Canada and the UK's Medicines and Healthcare products Regulatory Agency jointly published five guiding principles for predetermined change control plans for machine-learning-enabled medical devices: Focused and Bounded, Risk-based, Evidence-Based, Transparent, and Total Product Lifecycle (TPLC) Perspective. Under the first, a plan's characterisation can include “mechanisms to detect and revert or stop implementation of a change that fails to meet specified performance criteria”. That is step 5 of this process, written by an agency: bounded scope, declared validation, and a way back.

Cameras must detect people near forklifts with a 60-millisecond response budget and intermittent connectivity.