Model Packaging and Runtime Contracts

A ten-class classifier is exported to a portable runtime. The probability vector that comes back is a row of numbers. Nothing inside it says which column is which class. That mapping belongs to the fitted estimator, not to the array. scikit-learn's documentation says so for LogisticRegression.predict_proba: “Returns the probability of the sample for each class in the model, where classes are ordered as they are in self.classes_.” The same page adds that “The returned estimates for all classes are ordered by the label of classes”. The order is the model's, never the caller's.

A serialized model loads successfully and produces nonsense, because serving applies a different normalization and a different class order. The bytes are intact. The package omitted the assumptions required to interpret them.

Packaging converts learned state into an operational component. It should declare how inputs are validated, how preprocessing is bound, what outputs mean, which runtime is required, and how initialization or failure behaves.

The Python documentation is blunt about the format most weight files use. “The pickle module is not secure. Only unpickle data you trust.” It adds that malicious pickle data can execute arbitrary code during unpickling. scikit-learn's model persistence page repeats the warning: loading a pickle “can execute arbitrary code”, and “You should never load a pickle file from an untrusted source”. joblib says the same of its own loader: “arbitrary Python code can be executed when loading a serialized object with joblib.load()”. Three independent projects point at the format itself, not at one library that got it wrong.

There is an incident record behind the warning. Two malicious models sat on Hugging Face, stored as PyTorch pickle archives compressed with 7z instead of the default ZIP. The compression was the point: Hugging Face's Picklescan could not parse the archive. Inside was a platform-aware reverse shell to the hardcoded address 107.173.7.141, placed at the start of the pickle stream so that it executed before the reader reached the broken portion. ReversingLabs disclosed the pair as nullifAI on 6 February 2025. Hugging Face removed the models in under 24 hours, and Picklescan was updated to handle broken pickles. The disclosure states the mechanism plainly: “Pickle is considered an unsafe data format, as it allows Python code to be executed during ML model deserialization.” A model file from an unverified source is code, and it runs on load.

The platform-side scanner is not the contract either. Sonatype disclosed four CVEs in picklescan on 11 March 2025 — CVE-2025-1716, CVE-2025-1889, CVE-2025-1944 and CVE-2025-1945 — all addressed as of picklescan 0.0.23. This is the scanner Hugging Face runs over uploaded pickles. The GitHub Advisory Database rates the first of them CVSS 5.3 Moderate, published on 3 March 2025, affected versions up to 0.0.21, patched in 0.0.22. The flaw was that pip was not treated as an unsafe global, which let a model install a malicious PyPI package through pip.main(). Hugging Face's own pickle scanning documentation had already disclaimed the guarantee: “this is not 100% foolproof. It is your responsibility as a user to check if something is safe or not.” Provenance is part of the package contract. No scanner supplies it on your behalf.

The package should make its assumptions executable. Five layers have to travel together. The learned state: weights, trees, vocabularies, calibration or index snapshots. The transformation logic: required preprocessing, tokenization, postprocessing and class mapping. The interface contract: input and output schemas, batch semantics, errors and version compatibility. The runtime contract: framework, libraries, hardware capabilities, memory and initialization. The evidence metadata: training origin, validation scope, limitations, owner and artifact digest.

Every layer left outside the package becomes a thing that can move without the artifact noticing. The class vocabulary in the case above is the transformation layer taken out and stored somewhere editable.

Bundling preprocessing with the model reduces one class of skew. It also makes the feature logic harder to share and to inspect. Keeping transformations outside improves reuse, but only under strict compatibility contracts.

There is no universal boundary. The right package groups components that must change together, and exposes stable interfaces for the components allowed to evolve independently. The question a boundary answers is not aesthetic. It decides which pieces can be replaced by someone who never reads your training code, and which will break silently when they are.

The format should fit the runtime and the evidence requirements. A framework-native artifact preserves framework behavior and custom components. It is the fastest path for a known runtime, and it carries a large dependency surface whose custom code can be hard to audit. A portable graph or standard format targets multiple runtimes through a constrained operator set, at the price of conversion validation and fallbacks for unsupported operators. A custom serving package combines model, transformations, policies and optimized kernels for maximum control, and takes on a higher maintenance and security burden in exchange.

One of these boundaries has been audited in public, which makes it worth reading closely. EleutherAI, with Hugging Face and Stability AI, commissioned Trail of Bits to assess the safetensors library. The engagement ran from 20 to 24 March 2023 — two consultants, two person-weeks — and the final report was delivered on 3 May 2023. It recorded ten findings: 0 high, 3 medium, 3 low, 3 informational and 1 undetermined, nine of them in the category Data Validation and one in Patching. None of them was an arbitrary-code-execution flaw in the serialization format itself. Hugging Face and EleutherAI announced the result jointly on 23 May 2023: “No critical security flaw leading to arbitrary code execution was found.”

The interesting part sat immediately beside the format. Finding 8 was “PyTorch conversion utility is vulnerable to arbitrary code execution”. Finding 9 was “Python dependencies are not semantically versioned”. The bytes were cleared. The tool that produced them, and the dependency pins around it, were not. That is the packaging boundary this section is arguing about, drawn by an auditor rather than by a preference. Choosing a safe format buys you the format. Everything you bundled next to it still needs its own contract.

The contract should let a consumer integrate the package without reading training code. Five moves, in order.

1. Define input semantics — names, shapes, units, missingness, ranges and batch behavior. You do not have to invent the envelope. KServe's Open Inference Protocol (V2) already specifies six required HTTP endpoints: GET /v2/health/live, GET /v2/health/ready, GET /v2, GET /v2/models/<model_name>[/versions/<model_version>], GET /v2/models/<model_name>[/versions/<model_version>]/ready and POST /v2/models/<model_name>[/versions/<model_version>]/infer. The version segment in brackets is explicitly optional. NVIDIA Triton Inference Server implements that protocol rather than a private one: “Triton exposes both HTTP/REST and GRPC endpoints based on standard inference protocols that have been proposed by the KServe project.”

2. Bind transformations — package or version preprocessing, vocabularies and postprocessing. The ZipMap in the case above is the move: the label vocabulary goes into the exported artifact instead of staying an index convention.

3. Define outputs and errors — labels, score meaning, abstention, invalid input and timeout behavior. Note what the protocol above separates. A model-ready endpoint and a model-metadata endpoint are distinct from the inference endpoint, because "can this thing answer" and "what does this thing accept" are different questions from "here is a request".

4. Declare runtime needs — dependencies, hardware, memory, initialization and concurrency assumptions. An ambiguously resolved dependency name is a live failure mode of this step, not a hypothetical one. Between 25 and 30 December 2022, a malicious torchtriton package sat on PyPI under the same name as the one on PyTorch's own nightly index. The project's advisory is direct: “PyTorch-nightly Linux packages installed via pip during that time installed a dependency, torchtriton, which was compromised on the Python Package Index (PyPI) code repository and ran a malicious binary.” The binary read /etc/hosts, /etc/passwd, $HOME/.gitconfig, $HOME/.ssh/* and the first 1,000 files in $HOME. It sent them out over DNS queries to *.h4ck[.]cfd. ReversingLabs analysed the same attack on 4 January 2023 and added that the exfiltrated data was scrambled but not encrypted, and that the malicious package saw roughly 2,500 installations. The fix was a naming fix: PyTorch renamed the dependency to pytorch-triton and registered a dummy package on PyPI so the substitution could not recur. Five days, one unpinned name.

5. Validate compatibility — run semantic fixtures and resource tests in every supported target, not only in the one you built on.

A consumer should know what the model accepts, what it returns, which failures are expected, and which version combinations are supported. Hidden assumptions become production incidents.

Treat the inference contract as a versioned public interface even when the only consumer is another internal team.

MLflow turns that promise into a file layout. Its documentation calls an MLflow Model “a standard format for packaging machine learning models that can be used in a variety of downstream tools”. One saved model carries several flavors, and each flavor is what a particular tool knows how to read. Databricks documents the same convention. The convention travels with the artifact, which is the whole point of having one.

“Which version combinations are supported” is a solved shape too, not an aspiration. ONNX separates three versioned entities: IR version, operator-set (opset) version per domain, and model version. Its versioning document defines the middle one precisely: “An operator set represents a specific version of a domain, indicated by a pair (domain, version)”. A model declares the opsets it requires inside itself, in ModelProto.opset_import. The same document carries the released-versions table from ONNX 1.0 to 1.23.0.

The implementer answers with a window, not with a guarantee. ONNX Runtime's compatibility page states: “ONNX Runtime supports all opsets from the latest released version of the ONNX spec. All versions of ONNX Runtime support ONNX opsets from ONNX v1.2.1+ (opset version 7 and higher).” It publishes a table mapping each ONNX Runtime release to the ONNX version and opset it supports. That is what a version contract looks like when both sides write it down. A declaration carried inside the artifact, and a bounded support window published by whoever has to load it.