ML Platform Engineering, Self-Service, and Multi-Tenancy

One convenience link, published so that other people could use a model, handed out an entire storage account.

A central team builds one pipeline template for every model. Teams take it up at first, then fork it because time-series splits, streaming features, and human review do not fit the abstraction.

Platform engineering should remove repeated operational work while preserving the domain choices that determine evidence and risk. A paved road is valuable when it is easy to use, observable, and possible to leave deliberately.

Tenancy is where a shared platform earns or loses trust, and the documentation for shared clusters begins with quotas. The Kubernetes multi-tenancy documentation states that “By mapping tenants to namespaces, cluster admins can use quotas to ensure that a tenant cannot monopolize a cluster’s resources or overwhelm its control plane”, and that limits on object count “ensure fairness and aim to avoid noisy neighbor issues from affecting other tenants that share a control plane”. Google's best-practice guide for enterprise multi-tenancy on Kubernetes Engine gives the same two controls in the same order: “To provide a logical isolation between tenants that are on the same cluster, implement namespaces”, then “To ensure all tenants that share a cluster have fair access to the cluster resources, enforce resource quotas”.

Quotas decide how much of a cluster a tenant may take. They say nothing about what a tenant may reach. An ML platform can hold the first while losing the second.

On 4 April 2024 Wiz Research and Hugging Face jointly disclosed one such loss. A malicious pickle-serialised model, uploaded to the Hugging Face Inference API, allowed container escape and cross-tenant access to other customers' models on the shared inference infrastructure. Shir Tamari and Sagi Tzadik described the route in Wiz's write-up that day: “Wiz Research was able to compromise the service running the custom models by uploading our own malicious model and leveraging container escape techniques to break out from our tenant and compromise the entire service.” Hugging Face confirmed in its own post the same day that Wiz “identified shortcomings of our sandboxed compute environments by running arbitrary code within the system thanks to pickle”, and said all issues related to the exploit were resolved.

The upload was not an attack on the platform's edges. It was the platform's advertised function — run the model a user brings — carried one step further than the sandbox could hold.

Teams adopt platforms when they reduce time, cognitive load, and operational risk better than local alternatives. A mandate can leave teams adopting the platform in name only, while they bypass its controls through notebooks, side systems, or copied credentials.

Treat product teams as users: research workflows, onboarding, documentation, support, service objectives, deprecation, and feedback all matter.

DORA's 2019 report gives a platform the four numbers it should move. They are deployment frequency, lead time for changes, time to restore service, and change failure rate. Nothing about tooling appears in that list. The report finds that “elite performers deploy code 208 times more frequently than low performers”, that “the elite group has 106 times faster change lead times than low performers”, and that “elites have 2,604 times faster time to restore service than low performers”. The last two get swapped in retellings. The large multiplier belongs to recovery.

Five years later the same research programme measured the platforms themselves. DORA's 2024 report, released on 22 October 2024 with nearly 3,000 respondents, is the first to quantify internal developer platforms. Of those respondents, 89% were using one. Platform users showed 8% higher individual productivity, 10% higher team performance and 6% higher organizational performance, with developer independence adding a further 5% at both individual and team level.

Then the platform engineering chapter gives the other half of the result: “However, these gains do not come without some drawbacks. Throughput and change stability saw decreases of 8% and 14%, respectively, which was a surprising result.” The report links that instability to higher burnout. A platform can make its users feel faster while the two delivery metrics move the other way. That is why the second pair of numbers belongs on the platform dashboard beside the first. A satisfaction survey on its own is not evidence of a working platform.

The platform should expose stable capabilities without absorbing every domain decision.

The foundational layer is not a given. It has version numbers. NVIDIA's record for CVE-2024-0132, published on 26 September 2024, reads: “NVIDIA Container Toolkit 1.16.1 or earlier contains a Time-of-check Time-of-Use (TOCTOU) vulnerability when used with default configuration where a specifically crafted container image may gain access to the host file system.” It covers NVIDIA Container Toolkit v1.16.1 and earlier and GPU Operator up to 24.6.1. NVIDIA scored it CVSS 3.1 base 9.0 CRITICAL, vector AV:N/AC:L/PR:L/UI:R/S:C/C:H/I:H/A:H; NVD's own analysis of the same flaw scored it 8.3 HIGH. Wiz, which found it, estimated that the affected toolkit was present in over 35% of cloud environments.

Every layer drawn above the first one assumes a container boundary. That boundary is itself a patchable, versioned platform capability — with a default configuration, an affected range, and a number that somebody on the platform team owns.

A platform can provide common contracts without owning every implementation.

Spotify's paved road has a date, an origin story and a stated exit. A fragmented tooling ecosystem had produced what Spotify called “rumour-driven development”, and eight engineers built the first “Golden Path” tutorial during a Hack Week. Gary Niemen, a product manager there, described both in a post of 17 August 2020. The Golden Path has since been extended to client development, data engineering, data science, machine learning, web and audio processing, and is in Niemen's words “the 'opinionated and supported' path to 'build something'”.

What separates it from a mandate is written down by the operator: “If you are an adventurer you can of course leave the Golden Path and do your own thing, but then you will not have the same support.” The exit is named, and its price is support rather than permission.

DORA's 2024 report names Spotify among the industry leaders that popularised the practice, and defines golden paths as “highly-automated, self-service workflows that users of the platform use when interacting with resources required to deliver and operate applications”. That is the middle column of this comparison: an opinionated implementation, maintained, with a documented way out. It is not a standard interface, which fixes inputs and outputs and admits several implementations. It is not a mandate either, which fixes the implementation and pushes the mismatch into workarounds nobody records.

Begin with repeated user work and a measurable service promise.

Measure the promise in both directions. DORA's 2024 report found platform users reporting 8% higher individual productivity and 10% higher team performance. In the same report, throughput fell 8% and change stability fell 14%. A platform review that collects only the first pair of numbers will keep reporting success through the period in which delivery gets worse.

Teams will choose the shortest path to delivery. The platform's safe path must be easier than building untracked scripts, while still allowing honest domain variation. That is what the Golden Path buys with the word supported and gives back with the adventurer's exit.

A successful platform reduces repeated engineering and increases visibility without claiming ownership of the model's intended use or product harm. It is worth noticing that none of the four incidents in this lesson involved a team refusing the platform. A link published from Microsoft's own repository, a model uploaded to the Hugging Face Inference API, a training job running on SAP AI Core, and the default configuration of the NVIDIA Container Toolkit were all the supported path, used as designed. Evidence has to be cheaper than bypass. And the supported path has to be safe on the days when nobody bypasses it.