MUSE: Multi-Tenant Model Serving With Seamless Model Updates

AI summary

Overview

Research area: Machine learning systems and model serving infrastructure, specifically multi-tenant Score-as-a-Service platforms for fraud detection in financial systems.

Technical level: Intermediate. The paper combines production systems engineering (routing, Kubernetes deployments, latency SLOs) with statistical machinery (posterior correction, quantile mapping, Beta mixture fitting). Readers comfortable with classification scores, calibration, and service architecture will follow it; the math is presented compactly.

Scope: MUSE is a deployed model serving framework that lets a model provider retrain, swap, and promote models across many tenants without any client-side threshold recalibration.

What This Paper Is About

In fraud detection, a model's raw score is turned into an action (block, allow, review) by decision thresholds that each client sets and owns in their own infrastructure. When the provider retrains a model, the score distribution shifts, so the old thresholds no longer capture the intended fraction of risky events — and fixing that requires coordinating threshold changes across hundreds of clients. MUSE solves this by decoupling model scores from client decision boundaries: it transforms every predictor's output onto a fixed, stable reference distribution so that client thresholds keep meaning the same thing after a model update.

Key Contributions

1. A two-level score transformation combining Posterior Correction (to remove bias from majority-class undersampling during training) with Quantile Mapping (to force the final score onto a stable reference distribution). It requires no access to client thresholds and no labeled data.
2. An intent-driven model serving abstraction that treats routing, shadowing, and transformation updates as first-class deployment primitives, enabling safe, automated, zero-friction model promotion in multi-tenant real-time environments.
3. A graph-based resource reuse design where a single predictor (or ensemble member) can serve many tenants, so adding a model to an ensemble costs only the marginal difference in models rather than a full replica.
4. Results and operational insights from a real-world production deployment at Feedzai, processing thousands of events per second under strict latency and availability requirements.

Main Findings

• Stable scores are achievable without labels. Quantile Mapping uses only unlabeled score data, which matters because fraud labels are delayed, sparse, or absent in new deployments. The transformation is monotonic, so event ranking and predictive performance are preserved; only the score distribution changes.

• Raw scores are unusable for thresholds. In the cold-start case (an 8-model ensemble), untransformed scores were confined entirely to the first bin [0.0, 0.1[, giving a 43% relative error there and −100% error in every bin from [0.1, 1.0].

• The cold-start default transformation drifts. Predictor v₀ (default transformation) reached 207% relative error in bin [0.7, 0.8[ and peaked at 1691% in [0.9, 1.0]. The client-specific transformation v₁ cut the [0.7, 0.8[ error from 207% to 7.1%, with −1.5% error in [0.5, 0.6[ and 11% in [0.6, 0.7[.

• Replacing an expert without updating the transformation breaks alert rates. Predictor p₁.₅ (new 3-model ensemble paired with the old transformation) showed roughly 35% positive error in the first bin and consistently negative errors elsewhere, implying severe under-alerting for any threshold above 0.1%. Predictors p₁ and p₂ stayed close to 0% error across bins.

• Posterior Correction dramatically improves expert calibration. On in-distribution validation data: expert m₁ (β ≈ 18%) ECE fell from 4.97×10⁻³ to 7.67×10⁻⁴ (−84.6%) and Brier from 1.46×10⁻⁴ to 9.61×10⁻⁵ (−34.2%); expert m₂ (β ≈ 18%) ECE fell from 4.83×10⁻² to 8.63×10⁻³ (−82.2%) and Brier from 2.70×10⁻³ to 4.36×10⁻⁴ (−83.9%); expert m₃ (β ≈ 2%) ECE fell from 4.23×10⁻² to 8.46×10⁻⁴ (−98.0%) and Brier from 1.86×10⁻³ to 1.58×10⁻⁵ (−99.1%). On live client (out-of-distribution) data, the truncated table shows m₁ ECE dropping from 1.35×10⁻² to 2.29×10⁻³; the remaining live-data rows and percentage changes are cut off in the excerpt.

• Updates did not degrade latency. During the cold-start-to-custom transformation swap, latencies stayed strictly below 30ms at the observed percentiles (p99.99 and p99.5). A 15-minute warm-up procedure generated spikes up to 50 req/s per new pod so JIT compilation completed before live traffic arrived.

• Model quality improved with a 10-week faster rollout. Predictor p₂ achieved a 1.1 percentage-point increase in Recall at 1% FPR over p₁. Because Quantile Mapping preserves ordering, Recall was identical between p₁.₅ and p₂. Removing client-side threshold adjustments accelerated deployment by 10 weeks.

• Production scale. MUSE processes over a thousand events per second and over 55 billion events in the last 12 months across several dozens of tenants. The evaluation cluster averages 4,500 events per second under SLOs of 30ms p99 latency, 150ms p99.9, and 99.95% availability.

• Measured fraud impact. In fraud detection alone, MUSE analyzed over $1.8 billion in volume across 7 million transactions, preventing more than $1.7 million in confirmed fraud. These figures are based on the two clients that currently provide feedback labels.

Methodology in Plain English

The team built a serving layer that separates "what a model outputs" from "what a client threshold means."

First, raw model outputs are passed through Posterior Correction, a closed-form rescaling that undoes the inflation caused by undersampling the majority (negative) class during training. Each expert model gets corrected using its own undersampling ratio β, so that no single expert dominates the ensemble simply because its scores sit closer to the extremes.

Second, the corrected expert scores are combined by a configurable aggregation function — commonly a weighted average — whose weights can be tuned per client without retraining the underlying models.

Third, the aggregated score is passed through Quantile Mapping, which aligns the score's cumulative distribution to a fixed reference distribution. This is approximated as a piecewise linear function over N precomputed quantiles, evaluated in O(log N) time by binary search. The reference distribution is configurable; in imbalanced fraud settings a distribution concentrated near 0 with a long tail toward 1 gives clients finer granularity in the useful alert-rate range of roughly 0.1% to 1%.

Because each tenant's data distribution differs, source quantiles must be estimated per client–predictor pair. The authors derive a sample-size requirement of approximately n ≈ z²(1−a)/(δ²a) for alert rate a, relative error δ, and z-score z (e.g., z = 1.96 for 95% confidence). For low-volume or brand-new clients who lack that data, MUSE substitutes a Beta mixture model fitted to the predictor's training scores. Fitting minimizes a cost function over the first four raw moments, using a stochastic search repeated across N_trial runs, with the final fit selected by minimum Jensen-Shannon Divergence against the empirical distribution.

On top of this, requests are routed by business intent (tenant ID, geography, schema) rather than by model version. Teams declare scoring rules evaluated sequentially and shadow rules evaluated in parallel, so a request can be served live by one predictor while being mirrored to others for offline evaluation. Since routing is stateless, promotions use standard Kubernetes rolling updates, and a code warm-up subprocess exercises the hot path before pods accept traffic.

Why This Matters

Impact on research. The paper reframes score calibration as an infrastructure problem rather than a purely statistical one. It shows that distributional invariance across model versions — not just per-model calibration — is the property that matters operationally in multi-tenant settings, and it demonstrates that this can be achieved without labeled data.

Real-world applications:

• Fraud detection platforms that must update models frequently against adversarial, shifting attack patterns.
• Credit underwriting and application fraud systems where retraining cycles are constrained by regulatory false-positive limits.
• Any multi-tenant SaaS scoring product where decision thresholds live in customer-owned infrastructure.
• Migration from legacy scoring systems, since the reference distribution can be chosen to match an existing production system's score distribution.

Industry relevance. The paper quantifies the cost of the naive approach: manual threshold tuning across a growing client base leads to "model stagnation," where better models stay undeployed because upgrading costs too much coordination. MUSE reports reducing model lead time from weeks to minutes and saving millions of dollars in fraud losses and operational costs. Feedzai has MUSE in production serving several dozens of clients with a roadmap to scale to hundreds in a few months.

Future Directions

• Scaling the tenant count. The paper notes a roadmap from several dozens to hundreds of clients; whether the routing and transformation configuration model stays manageable (or needs heavier automation) at that scale is not resolved.

• Improving beyond undersampling correction. The authors explicitly state that Posterior Correction removes only undersampling bias, not calibration distortion from model architecture choices, and that Platt Scaling or Isotonic Regression were ruled out because they require large amounts of labeled data. Alternatives that work under label scarcity remain open.

• Cold-start accuracy. The Beta-mixture prior is a stand-in until enough live unlabeled data accumulates. How quickly it should hand off to a fitted transformation, and how that handoff should be validated, is not fully specified.

• Detecting distribution drift after deployment. MUSE relies on shadow scoring and offline evaluation to validate a candidate before promotion. The paper does not report an automated mechanism for deciding when a live predictor has drifted enough to warrant re-estimating source quantiles.

Target Audience

Machine learning platform and infrastructure engineers building multi-tenant scoring services; applied scientists working on calibration for imbalanced, label-scarce domains like fraud; and technical leaders at financial institutions or vendors evaluating how to decouple model deployment from customer-side threshold management. Readers primarily interested in novel model architectures will find little here — the contribution is systems and deployment methodology.

Authors’ abstract

In binary classification systems, decision thresholds translate model scores into actions. Choosing suitable thresholds relies on the specific distribution of the underlying model scores but also on the specific business decisions of each client using that model. However, retraining models inevitably shifts score distributions, invalidating existing thresholds. In multi-tenant Score-as-a-Service environments, where decision boundaries reside in client-managed infrastructure, this creates a severe bottleneck: recalibration requires coordinating threshold updates across hundreds of clients, consuming excessive human hours and leading to model stagnation. We introduce MUSE, a model serving framework that enables seamless model updates by decoupling model scores from client decision boundaries. Designed for multi-tenancy, MUSE optimizes infrastructure re-use by sharing models via dynamic intent-based routing, combined with a two-level score transformation that maps model outputs to a stable, reference distribution. Deployed at scale by Feedzai, MUSE processes over a thousand events per second, and over 55 billion events in the last 12 months, across several dozens of tenants, while maintaining high-availability and low-latency guarantees. By reducing model lead time from weeks to minutes, MUSE promotes model resilience against shifting attacks, saving millions of dollars in fraud losses and operational costs.