The Pulse
Hugging Face Proposes Model Integration Contracts for Testing
Hugging Face has proposed Model Integration Contracts, small executable fixtures that let Transformers test model behavior on CPU during pull requests. The proposal follows a silent UMT5 attention bug that escaped detection for nearly two m
AI.info Team · Written with AI · fact-checked ·
“The LLM drafts; it doesn't approve.”
Tarek Ziadé, author of Hugging Face’s post
Hugging Face proposed Model Integration Contracts on November 3, a way to test model behavior in Transformers pull requests without loading full-size checkpoints. The contracts pair a reduced model fixture with a runnable example and reviewed expected outputs, aiming to catch regressions on inexpensive CPU runners before code reaches the main branch. The proposal grew out of a bug that let a text model attend to future tokens during training without producing an obvious failure.
A UMT5 regression passed ordinary checks
On July 30, Transformers enabled PyTorch’s scaled dot-product attention for T5. The change also let UMT5’s decoder see future tokens—the answers it was meant to predict—during training. Nothing crashed, and the fault remained in the main code branch until a fix on September 28.
Hugging Face says a small UMT5 fixture running on CPU flagged the problem when the team replayed earlier code commits. The reported differences from the reviewed baseline were 1.22 in label log-probabilities and about 0.20 in training loss. The test needed to use the scaled dot-product attention path: running only the eager attention variant missed the bug.
The incident highlights a gap between tests that confirm a model runs and tests that check whether it still behaves as expected. Transformers supports more than 750 model types, and the project’s post counts over 88,000 CPU tests on its main branch and more than 200,000 GPU tests in a daily run. Running every full-checkpoint integration test on every proposed code change would require sustained GPU capacity, so many of those checks happen later.
Each contract pairs a fixture with a real task
A Model Integration Contract, or MIC, has three parts: a smaller fixture built from a real checkpoint’s configuration, a runnable Markdown example of a model task, and baseline outputs that a person has reviewed. The fixture preserves the model’s distinct layer types and keeps tokenizer and processor files unchanged. If the example no longer produces the recorded baselines, the contract fails.
The examples are meant to exercise more than model loading and a forward pass. Hugging Face’s SmolLM example generates tokens from a prompt and compares them with recorded token IDs; its checks also cover training loss, finite gradients and whether an optimizer step changes the weights. A dataset on the Hub pins each contract to a fixed revision and records the package versions used to produce its baselines, giving CI runs a reproducible reference.
In an experimental UMT5 case, the fixture contained 66 megabytes of weights, compared with 1,227 megabytes for the source checkpoint. After reverting the fix on a proof branch, the selected T5 contract failed as expected; its runner step took 13.4 seconds on CPU, excluding setup. Hugging Face says fixtures are a quick signal for pull requests, not a substitute for tests on full checkpoints.
Hugging Face leaves the CI check non-blocking
The project’s tmic command-line tool can draft a contract and build its fixture. A language model drafts the usage code and manifest, but people review changed expectations before approval. Hugging Face says its local workflow stops before uploading anything; generating the initial draft for a new model uses Inference Providers and is billed to the user’s account.
The proposal’s limits are still being measured. As of November 1, the MIC job in Transformers CI reported failures but did not block pull requests. Hugging Face says it still needs to establish how often the checks catch real bugs in incoming changes, how often they produce false alarms and how much time baseline reviews require. Coverage across model families and common usage paths is also partial.
A separate Whisper experiment gives a reason for caution about what small fixtures can prove. Fixtures with random weights caught 18 of 24 injected faults; fixtures trained on a few examples caught all 24, but those were synthetic faults, and both approaches missed an assisted-decoding bug that appeared with the real Whisper checkpoint. Hugging Face’s proposal therefore adds an earlier test layer, while leaving full-checkpoint testing—and the evidence needed to decide whether MIC failures should block a pull request—in place.