Skip to content
AI.info

MLOps

Edge and On-Device MLOps

Manage packaging, fleet rollout, telemetry, privacy, fallback, and retirement for models running on heterogeneous edge devices.

By the end you can

Example

A content update shipped to the whole fleet at once

On 19 July 2024 CrowdStrike pushed a Rapid Response Content update — a new version of Channel File 291 — to Windows sensors. It went out with no staged rollout.

  • No staging: The Rapid Response Content update, a new version of Channel File 291, reached Windows sensors without a staged rollout, so the first cohort to run it was the fleet.
  • Blast radius: Microsoft estimated the update crashed 8.5 million Windows devices — under one percent of all Windows machines, which is what one percent of a large enough population looks like.
  • Not a model, still a release: The thing that shipped was content, not new sensor code, and it moved through the fast channel that content updates are given precisely because they are considered small.
  • The remedy is a rollout policy: CrowdStrike's root cause analysis of 6 August 2024 committed to canary testing and to successive deployment rings with bake-in time between them.
  • And a hand on the tap: It also committed to giving customers control over when content is delivered, which is the admission that propagation speed is itself a risk setting, not a convenience.

Deployment succeeds in the lab and fails in the fleet

A vision model meets latency and accuracy targets on the reference device. In the field, older cameras have different sensors, thermal throttling reduces throughput, and intermittent connectivity prevents timely updates.

Edge MLOps operates a fleet, not one benchmark device. Hardware, firmware, environment, battery, network, and physical wear all become release dimensions. The difficulty is not that any single device is unusual. It is that a release lands on all of them, in whatever state they happen to be in. In a population that size, a small failure rate stops being small.

Visual

The edge release envelope

An on-device model depends on a local stack that changes outside the ML team.

Fleet control is the layer that decides how fast the other four learn about a mistake. Cohorts, update channel, staged rollout, rollback, attestation, and expiry are all settings. Shipping with them off is a choice someone made.

FigureLayers · 5 layers
  1. 01

    Physical input

    Sensor calibration, optics, microphones, motion, environment, and wear.

  2. 02

    Device substrate

    Chipset, memory, power, thermal state, firmware, and operating system.

  3. 03

    Runtime package

    Model, kernels, preprocessing, postprocessing, and resource policy.

  4. 04

    Fleet control

    Cohorts, update channel, staged rollout, rollback, attestation, and expiry.

  5. 05

    Evidence return

    Health summaries, sampled outputs, incidents, and privacy-preserving telemetry.

Comparison

Cloud and edge operations fail differently

The same release discipline applies, but the control surface changes. Centrally, a bad version is withdrawn by the operator. On devices it is withdrawn only as fast as devices come back and accept a replacement. That is a property of the population, not of the operator's intent.

FigureComparison · 3 columns

Central serving

Infrastructure and telemetry are concentrated in a managed environment.

  • Rapid rollback is possible
  • Capacity can be pooled
  • Dependencies are observable
  • Network is required for each request

On-device inference

Execution is distributed across heterogeneous and intermittently connected devices.

  • Low latency and local privacy
  • Slow or partial update propagation
  • Limited telemetry and storage
  • Hardware-specific behavior matters

Hybrid design

Local inference uses cloud support for updates, verification, or complex fallback.

  • Balances latency and capability
  • Needs explicit offline behavior
  • Must handle version skew
  • Example: local detection with cloud review for uncertain cases

The device population is part of the evaluation set

Identical model weights can behave differently across chipsets, accelerators, camera pipelines, locales, or power modes. Segment the fleet by observed compatibility and risk, not only by marketing model names.

How unevenly software reaches a device population has been measured. The FTC published the numbers in February 2018. The cheapest devices received an average of 1.3 years of security-update support, mid-tier devices 1.4 years, and the most expensive 1.9 years. Individual devices ranged from about half a year to more than three, and one expensive device received no updates at all. The same device model could get 1.3 years of support through one carrier and 2.5 years through another. The report's preliminary findings state it in two sentences: “Support periods and update schedules are highly variable. Formal support policies are rare.”

The evidence came from section 6(b) orders issued in May 2016 to Apple, Blackberry, Google, HTC, LG, Microsoft, Motorola and Samsung. The FCC ran the parallel inquiry into carriers, announced 9 May 2016.

Read the carrier line again. Identical hardware, identical model name, two different support lifetimes depending on who sold it. A cohort key built from the marketing name would have merged those two populations into one row.

Release evidence should therefore include device classes, runtime versions, thermal conditions, memory pressure, and sensor characteristics. The rollout policy can then target compatible cohorts and preserve a rollback package.

Case

Integer-only inference, benchmarked on the phones it shipped to

A model small enough to run on a phone has given something up, and the amount can be put in numbers before it ships. One scheme runs inference “using integer-only arithmetic”, and a convolution under it “involves 8-bit integer operands and a 32-bit integer accumulator”. Jacob and colleagues published it in 2018. Training simulates the quantization, so the accuracy loss is measured before shipping rather than found after. The benchmarks name the hardware: Snapdragon 835 LITTLE and big cores, and Snapdragon 821 big cores. A quantized model is a different model in the field. Its accuracy has to be measured on the fleet, not on the reference handset sitting on somebody's desk.

Steps

Release to a device fleet

Treat compatibility and propagation as first-class release risks.

Step 2 — validating the package per cohort — has an industry answer worth copying. MLPerf Tiny, published in 2021, is the first industry-standard benchmark suite for ultra-low-power inference, assembled with more than 50 organisations behind it. It covers four tasks: keyword spotting, visual wake words, image classification, anomaly detection. Its design principle is the one an edge release board needs, and the abstract states it: “MLPerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems.” Quality targets are fixed. Latency and energy are then measured on the hardware actually submitted, rather than estimated from it. Even so, the expensive measurement is the one that gets skipped. The v1.0 round, published on 9 November 2022, carried 59 performance results from 8 organisations. Only 39 of them came with an energy number.

Step 3 — staging the rollout — is the step the case above paid for. CrowdStrike's own root cause analysis of 6 August 2024 sets the rule out as finding 6, “Template Instances should have staged deployment”: “New Template Instances that have passed canary testing are to be successively promoted to wider deployment rings or rolled back if problems are detected.” Nothing in that sentence is difficult or novel. It was written after 19 July 2024, which is the usual date on this kind of policy.

FigureProcess · 5 steps

  1. 1. Define device cohorts

    Group by hardware, firmware, sensors, runtime, and observed behavior.

  2. 2. Validate the package

    Test latency, memory, energy, thermal behavior, and quality per cohort.

  3. 3. Stage the rollout

    Use signed packages, small cohorts, soak time, and automatic stop rules.

  4. 4. Preserve recovery

    Keep a rollback or safe baseline and define offline expiry behavior.

  5. 5. Monitor minimally

    Collect cohort-level health and targeted evidence without exporting unnecessary raw data.

Key idea

Telemetry can recreate the privacy problem edge inference was meant to solve

Running locally may keep raw inputs off the network. Uploading detailed samples, embeddings, logs, or traces puts sensitive information back on it. Telemetry should be minimized, aggregated, sampled, access-controlled, and justified by a specific operational question.

Apple's Siri quality-evaluation programme is the worked example, and the important detail is that the local part worked as designed: wake-word detection ran on the device. What came back was audio. Human reviewers listened to less than 0.2 percent of Siri audio requests. Apple's own account of it: “Before we suspended grading, our process involved reviewing a small sample of audio from Siri requests — less than 0.2 percent — and their computer-generated transcripts, to measure how well Siri was responding and to improve its reliability.” After the July 2019 disclosure Apple suspended grading. On 28 August 2019 it stopped retaining audio by default, made retention opt-in, and limited listening to Apple employees. The related class action, Lopez v. Apple, settled for a $95,000,000 non-reversionary common fund, plus a commitment to confirm permanent deletion of Siri audio recordings collected before October 2019. That is the price of a sampling rate below one in five hundred, chosen for a legitimate quality question.

Device identifiers and error reports can also become tracking data. Privacy review belongs in the observability design.

Federated averaging was proposed for exactly this trade. McMahan and colleagues left the training data on the mobile devices and aggregated locally computed updates instead. Their 2017 paper reports “a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent”. Fewer rounds is also less exposure. Neither property is free. What leaves the device stays a design decision, never a default.

Local inference improves privacy only when the evidence returned from the device is also controlled.

Fleet state is eventually consistent

Rollout can take weeks, and for all of those weeks devices run several versions. Product logic, support tools, and server APIs must tolerate that skew.

A filed safety recall shows the shape of it. Tesla filed NHTSA recall 23V-838 on 12 December 2023, covering 2,031,220 vehicles, and the remedy is software: “At no cost to customers, affected vehicles will receive an over-the-air software remedy, which is expected to begin deploying to certain affected vehicles on or shortly after December 12, 2023, with software version 2023.44.30. Remaining affected vehicles will receive an over-the-air software remedy at a later date.” A recall of two million units therefore has a start date and no completion date. For whatever interval follows, the fleet is two fleets.

Shipping is also not the same as fixing. Consumer Reports' preliminary evaluation of the delivered update found it insufficient: its testers still engaged and used Autopilot after covering the in-car driver-monitoring camera. Evidence about a remedy has to come back from the field, not from the release note.

Retirement requires more than removing a registry entry. Revoke signing trust if needed, expire packages, stop compatible APIs, and verify that unsupported devices fall back safely. In the UK this is now statutory rather than an internal good intention. SI 2023/1007, made 14 September 2023 and in force 29 April 2024, requires a manufacturer of connectable consumer products to publish a “defined support period”. Regulation 2(1) defines that as “the minimum length of time, expressed as a period of time with an end date, for which security updates will be provided”. Schedule 1, paragraph 3(6) closes the exit: “The security requirements in this paragraph are not met if the defined support period is shortened after the publication of the information in sub-paragraph (2).” The date can move outward and not inward. The same obligation is provision 5.3-13 of ETSI EN 303 645, which the SI names as a route to deemed compliance.

Key takeaways