Skip to content
AI.info

Implementation Guides

CI/CD for Machine Learning Projects

How to implement continuous integration and deployment for ML systems. Covers testing strategies, model validation gates, automated retraining, and deployment pipelines.

CI/CD for Machine Learning Projects

Gabriele Masetti ·

Why ML CI/CD Is Not Software CI/CD

Traditional CI/CD versions and tests one artifact: code. A pull request triggers a build, a test suite, and a deploy. Machine learning systems have three artifacts that change independently and interact unpredictably: code (training scripts, feature transforms, serving logic), data (training sets, feature stores, reference distributions), and models (weights, hyperparameters, architecture). A green test suite on the code doesn't tell you anything about whether the model trained on this week's data still performs above threshold.

This is why MLOps teams talk about pipelines rather than pipelines-plus-an-afterthought: the training pipeline itself is a build artifact that has to be versioned, tested, and reproducible, not just the code that defines it.

Three consequences follow directly from this:

The Three Layers to Version

Before building pipelines, decide how you'll version each layer — retrofitting this later is painful.

  1. Code: Git, as usual. Training scripts, preprocessing, serving code, infra-as-code.
  2. Data: DVC is the most widely adopted open-source option — it stores pointers to data in Git while the actual files live in S3, GCS, Azure Blob, or a local remote, and dvc.yaml encodes the pipeline stages (prepare → train → evaluate) as a DAG with cached, content-hashed outputs. dvc repro re-runs only the stages whose inputs changed, giving you Makefile-style incrementality for ML pipelines.
  3. Models + experiments: MLflow Tracking logs parameters, metrics, and artifacts per run; MLflow Model Registry (or the Weights & Biases Registry) then holds registered model versions with aliases such as champion and challenger — replacing the older stage labels (Staging/Production), which MLflow deprecated as of version 2.9.0 in favor of mutable, promotable aliases; aliases are still the documented mechanism in MLflow 3, the current major line since 11 June 2025. W&B's Registry does the same thing with artifact aliases and can fire a webhook to kick off a deployment pipeline when a new version is promoted.

A minimal dvc.yaml capturing a training stage looks like this:

stages:
  prepare:
    cmd: python src/prepare.py --input data/raw --output data/processed
    deps:
      - src/prepare.py
      - data/raw
    outs:
      - data/processed

  train:
    cmd: python src/train.py --data data/processed --out models/model.pkl
    deps:
      - src/train.py
      - data/processed
    params:
      - train.learning_rate
      - train.n_estimators
    outs:
      - models/model.pkl
    metrics:
      - metrics/train_metrics.json:
          cache: false

  evaluate:
    cmd: python src/evaluate.py --model models/model.pkl --data data/processed
    deps:
      - src/evaluate.py
      - models/model.pkl
    metrics:
      - metrics/eval_metrics.json:
          cache: false

dvc.lock records the exact hashes of every dependency and output, so dvc repro gives you a build cache: if data/raw and src/prepare.py are unchanged, DVC skips prepare and reuses the cached output. That's the closest ML equivalent to a software build cache, and it's what makes CI runs fast enough to be practical.

Concrete Pipeline: What Runs on Every Push

A realistic CI pipeline for an ML repo, wired into GitHub Actions, generally runs these stages in order and fails fast on the cheap checks first:

  1. Lint and unit tests — standard pytest / ruff / black on the code that isn't data-dependent (transform logic, config parsing, API contracts for the serving layer).
  2. Data validation — run GX Core (or an equivalent) against the incoming or refreshed training data. Checks include schema conformance, null-rate thresholds, category cardinality, referential integrity, and distribution sanity checks (e.g., a numeric column staying within historical min/max bounds). This step should block the pipeline before any compute is spent training. GX Core 1.0 rewrote the API and support for the older 0.18 line has ended, so a checkpoint script inherited from a pre-1.0 project needs porting before it will run at all.
  3. Pipeline execution via DVC — dvc repro (or triggering the equivalent stage in Airflow/Kubeflow/Dagster/Prefect) to reproduce the training pipeline with the exact code+data combination.
  4. Model evaluation against a baseline — compare the new model's metrics against the currently registered production model, not just an absolute threshold. This catches slow degradation that a fixed threshold would miss.
  5. Model tests — behavioral tests beyond aggregate metrics: invariance tests (prediction shouldn't change under label-irrelevant perturbations), directional expectation tests (increasing a known-positive feature shouldn't decrease a positive prediction), and minimum-functionality tests on a small curated slice, following the pattern popularized by Microsoft's "ML Test Score" and Ribeiro et al.'s CheckList methodology.
  6. Report generation — CML (Continuous Machine Learning, from Iterative, the makers of DVC) posts a comment on the PR with metrics, plots, and a diff against the base branch, so a reviewer sees model performance the same way they'd see a code diff. CML still works, but it is quiet — the most recent maintenance notice pinned in its README dates from 2023 — so treat it as a stable dependency rather than an actively developed one.
  7. Registration — if all gates pass and the branch is main, register the model version in MLflow or W&B Registry with a challenger alias, not directly to champion.
  8. Deployment (a separate, often manually or metric-gated step) — promote challenger to champion and roll out via canary or shadow deployment.

A trimmed GitHub Actions workflow implementing steps 1–4 and 6:

name: ml-ci

on:
  pull_request:
    branches: [main]

jobs:
  validate-and-train:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Lint and unit tests
        run: |
          ruff check src/
          pytest tests/unit -q

      - name: Data validation (GX Core)
        run: python scripts/run_gx_checkpoint.py --checkpoint training_data_checkpoint

      - name: Pull data and reproduce pipeline
        uses: iterative/setup-dvc@v1
      - run: |
          dvc pull
          dvc repro
          dvc push

      - name: Compare against production baseline
        run: python scripts/compare_to_baseline.py --new metrics/eval_metrics.json

      - name: Generate CML report
        uses: iterative/setup-cml@v2
      - env:
          REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          echo "## Model evaluation" > report.md
          cat metrics/eval_metrics.json >> report.md
          cml comment create report.md

Continuous Training (CT)

CT is the ML-specific extension of CD: instead of (or in addition to) deploying new code, the system automatically retrains and redeploys the model itself. Two common triggers:

For orchestration, Airflow 3.x, Dagster, Kubeflow Pipelines, and Prefect 3 are the current mainstream choices; Dagster's asset-centric model (defining the data/model products a pipeline should produce, and letting the scheduler figure out what needs rematerializing) fits CT particularly well because "is this model stale relative to its inputs" is exactly the kind of freshness question Dagster is built to answer. Kubeflow Pipelines remains the default when training already runs on Kubernetes.

Whichever orchestrator runs CT, it should call the same DVC/registry-backed pipeline used in CI — the goal is that a scheduled retrain and a PR-triggered CI run execute identical code paths, just with different triggers.

Model Registry as the Handoff Point

The registry is the seam between the training pipeline and the serving/deployment pipeline. Treat it as the source of truth for "what's allowed to be in production," not just a metadata store:

Deployment Strategies: Canary and Shadow

Deploying a new model version carries a different risk profile than deploying new code — the failure mode is often silent (wrong predictions, not crashes) — so gradual rollout strategies matter more here than in typical web CD.

Rollout stage Traffic to new model
1 5%
2 25%
3 100%

For teams not running Kubernetes-native serving, BentoML and cloud-native equivalents (SageMaker endpoints with production variants, Vertex AI traffic splitting) provide comparable canary/shadow primitives without requiring a full KServe/Seldon/Istio stack.

Drift Monitoring in Production

Once deployed, the model needs monitoring beyond infrastructure metrics (latency, error rate). Evidently AI is the most widely used open-source tool for this: it runs 20+ statistical tests (population stability index, Kolmogorov-Smirnov, Jensen-Shannon divergence, and others depending on feature type) to detect data drift (input feature distributions shifting) and prediction/target drift (output distribution or, once labels arrive, actual model performance shifting), and it produces both interactive reports and machine-readable test suites you can gate a pipeline on.

The practical pattern: schedule an Evidently drift check (daily or per-batch) comparing current production inputs against the reference dataset used at training time. If drift exceeds a configured threshold on key features, either alert on-call or automatically trigger the CT pipeline described above. Log predictions and, where available, delayed ground-truth labels so performance drift (not just input drift) is measurable — input drift without performance degradation is common and not always actionable.

Reproducibility Checklist

Everything above only works if a training run can be reproduced exactly. Concretely, that means pinning:

A model version registered without all five of these traceable is, in practice, unreproducible and should be treated as a liability rather than a deployable asset — you can't debug a production incident, audit a regulatory request, or safely roll back to it if you can't reconstruct exactly how it was produced.

Putting It Together

The organizing principle is that code, data, and model changes each need their own versioning and validation, but they converge on one pipeline definition (DVC stages or an orchestrator DAG) and one handoff point (the model registry) before deployment. CI validates data and code cheaply and fast; the training pipeline reproduces the model deterministically; the registry gates and records promotion; canary or shadow deployment limits blast radius; and drift monitoring closes the loop back into continuous training.

None of these pieces is exotic — DVC, MLflow, GX Core, Evidently and KServe are mature, widely deployed, open-source tools, and Seldon Core is the same class of tool under a licence you have to pay for in production — but the discipline of wiring them into one pipeline, rather than treating model training as a manual side process bolted onto ordinary software CI/CD, is what separates ML systems that stay reliable in production from ones that quietly rot.

Explore

More articles