Partial VOROS: A Cost-aware Performance Metric for Binary Classifiers with Precision and Capacity Constraints

AI summary

Overview

Research area: Machine learning evaluation metrics — specifically cost-sensitive performance metrics for binary classifiers, applied to clinical early-warning/alert systems.

Technical level: Advanced. The paper is built around ROC-space geometry, polygon feasible regions, lemma/proof derivations, and a 3D ROC surface integral, though the motivation and experimental results are accessible to applied ML practitioners.

Scope: The paper introduces Partial VOROS, a cost-aware performance metric for binary classifiers that is restricted to the region of ROC space satisfying a minimum precision constraint and a maximum capacity constraint on the number of predicted positives.

What This Paper Is About

Conventional ROC analysis cannot simultaneously handle three deployment needs that matter in high-stakes settings such as hospital alert systems: asymmetric costs for false positives versus false negatives, a minimum acceptable precision (to avoid alarm fatigue), and a hard cap on how many positive predictions the responding staff can handle. The authors extend the earlier VOROS metric (Ratigan and Cowen, 2025) — which lifts the 2D ROC curve into a 3D surface with a cost axis — so that it accounts for precision and capacity limits, producing a metric they call Partial VOROS (PV). The goal is a metric that correctly ranks classifiers when costs, performance constraints, and capacity constraints all matter.

Key Contributions

1. Characterization of a feasible region in ROC space. The authors show that the subset of classifiers meeting a minimum precision bound (Prec ≥ α) and a maximum capacity bound (N_TP + N_FP ≤ κ) forms a polygon in ROC space, and they establish its geometry exhaustively as three cases depending on κ: a triangle that excludes (0,1), a quadrilateral that includes (0,1), or a right triangle that includes (0,1) and in which the capacity constraint becomes redundant.

2. Definition of the partial area of lesser classifiers. They extend the "area of lesser classifiers" of Ratigan and Cowen (2025) to only count the portion of the feasible region consisting of classifiers that are worse at a given cost parameter t, and they prove this area takes the form of a sum of quotients of linear functions of t (Lemma 19).

3. Definition of Partial VOROS (PV). They average the normalized partial area over a user-specified distribution p(t) of the fractional cost parameter, giving PV(F) = (1/A*) ∫₀¹ p(t) max_τ (A*_t(F_τ)) dt, a higher-is-better metric bounded between 0.0 and 1.0 regardless of dataset or constraints.

4. Empirical demonstration on real clinical data. They re-analyze published early-warning-score ROC curves and run model-selection experiments on MIMIC-IV and eICU mortality prediction, showing PV can outperform cost-unaware and constraint-unaware alternatives at ranking classifiers for in-hospital alerts.

Main Findings

• Precision and capacity are linear constraints in ROC space. Precision ≥ α is equivalent to y ≥ [α(1−p)/((1−α)p)]·x, a line through the origin called the minimum precision line ℓ_α. The capacity bound N_TP + N_FP ≤ κ is equivalent to y ≤ (1/|P|)(κ − |N|x), a line of fixed slope −|N|/|P| called the maximum capacity line ℓ_κ. Because both depend on the raw sizes of the positive and negative subsets, the feasible region's area depends explicitly on class counts that must be fixed and known before evaluation.

• The feasible region has exactly three shapes under stated practical assumptions. Under |P| < |N|, p < α < 1.0, 0 < κ < |D|, and a bound on t, the region is a triangle excluding (0,1) when 0 < κ < |P|; a quadrilateral including (0,1) when |P| ≤ κ < (1/α)|P|; and a right triangle including (0,1) when (1/α)|P| ≤ κ < |D|, where capacity becomes redundant and precision alone dominates. Figure 2 illustrates these with |P| = 1000, |N| = 9000 (p = 0.1) and α = 0.15 for κ = 900, 3000, and 9100.

• PV is cost-monotone, dominance-respecting, and fast to compute. Lemma 21 shows that if one ROC curve lies at or above another for all false positive rates, its VOROS and partial VOROS are higher. The maximum over thresholds is computable in O(h_c) time from the convex hull of a ROC curve, and only one area calculation is needed per value of t.

• Each candidate early warning score can win under some constraints (Edelson et al. re-analysis). Using ROC curves measured on 362,926 patient stays across 7 hospitals in Connecticut (outcome: transfer to ICU or death within 24 hours; prevalence p = 4.6%), the authors find NEWS2 performs best at high α and low capacity, the Rothman Index has superior recall at lower α and low capacity, and NEWS performs best once capacity exceeds 5% of all examples. Edelson et al. (2024) had previously concluded NEWS "outperformed" the other two scores; PV analysis adds nuance.

• PV beats or ties alternatives on MIMIC-IV in the permissive scenario. In scenario 1 (α = 0.15, κ = 0.5|D|, C_FP:C_FN ~ Unif(1/9, 1/6)), average test cost on MIMIC-IV was 0.261 for max PV and max VOROS, versus 0.303 for max pAUROC and 0.302 for max recall. On eICU, max PV and max VOROS both achieved 0.319, versus 0.336 for the other two strategies.

• Ignoring constraints hurts in the stringent scenario. In scenario 2 (α = 0.5, κ = 0.1|D|, C_FP:C_FN ~ Unif(1/40, 1/20)), max VOROS — which is unaware of constraints — yielded the worst cost on both datasets: 0.640 on MIMIC-IV versus 0.538 for the other three strategies, and 0.772 on eICU versus 0.707. The authors note this highlights the inadequacy of ignoring constraints in model selection.

• Constraint satisfaction can fail to transfer from validation to test. On MIMIC-IV, all selected models satisfied the α and κ constraints on test data. On eICU, several models that satisfied the constraints on validation fell slightly short on test (for example, precision of 0.44 instead of 0.5), attributed to statistical variation across subsets of the same size.

Methodology in Plain English

The authors start from the ROC curve, where each thresholded classifier is a point with false positive rate on the x-axis and true positive rate on the y-axis. They define a single fractional cost parameter t that captures both the ratio of false-positive to false-negative costs and the relative sizes of the positive and negative classes, so that a point's "normalized cost" is t·h + (1−t)(1−k), ranging from 0.0 at the perfect point (0,1) to 1.0 at the worst point (1,0).

Next they translate the two operational constraints into lines. Precision ≥ α becomes a line through the origin; capacity N_TP + N_FP ≤ κ becomes a line with fixed negative slope. The part of ROC space above the precision line and below the capacity line is the feasible region, and they enumerate the possible polygon shapes and compute their areas with the shoelace formula.

For scoring, they borrow the idea of an "iso-performance line" — the set of ROC points with the same cost as a given point at a given t — and define a classifier's partial area of lesser classifiers as the portion of the feasible region lying below that line. Averaging this area (normalized by the area of the whole feasible region, and maximized over thresholds) across a user-supplied distribution over t gives Partial VOROS.

For experiments, they re-analyze published ROC curves from a multi-hospital deterioration study, then run their own model-selection comparison on MIMIC-IV (15,474/7,802/7,861 patient-stays in train/validation/test, prevalence .104/.103/.107) and eICU (38,532/19,267/19,267 ICU-stays, prevalence 0.116/0.114/0.114). They featurize 13 time-varying univariate channels with 7 summary functions over 3 windows, build logistic regression, multi-layer perceptron, and random forest classifiers via sklearn, and grid-search 24–80 hyperparameter configurations each, totaling over 140 ROC curves. They then compare four selection strategies — max pAUROC, max recall, max VOROS, and max PV — selecting both a curve and a threshold on validation data and reporting the resulting expected cost on test data.

Why This Matters

Impact on research: PV gives evaluation researchers a principled way to bake deployment constraints into a ranking metric, rather than checking constraints after the fact. It can be viewed as a 3D extension of partial AUROC that respects both capacity and precision, and it addresses a gap the authors identify in prior cost-sensitive and partial-area work.

Real-world applications:

• Hospital early-warning alert systems — the motivating case, where alarm fatigue and limited staff capacity are well-documented safety and operational concerns. A survey of critical care physicians in South Korean hospitals found that too many false positives were the top concern, with a median requested precision of at least 28.5%.
• Fraud detection — the authors name this as a domain where precision and capacity constraints are critical.
• Credit scoring — likewise named as a constraint-sensitive application.
• Information retrieval / document triage — the paper notes that there may be limited time to handle relevant documents, making capacity an absolute cutoff rather than a rate.

Industry relevance: Any deployed ML alerting or screening system with a human response team faces exactly this problem — finite responder capacity plus a minimum tolerable precision. PV provides a single number that reflects both the operating constraints and the business cost structure, which supports model selection and threshold selection that is aligned with deployment realities rather than with benchmark conventions.

Future Directions

• Handling edge cases and assumptions. The main paper restricts itself to practical assumptions (negatives more common than positives, p < α < 1.0, non-trivial capacity, a bounded cost parameter); the authors defer remaining edge cases to an appendix, which leaves open how general the full framework is in every regime.
• Choosing and justifying the cost distribution. The metric requires a distribution p(t), or equivalently a distribution over the cost ratio C_FP:FN. The paper offers uniform distributions on task-relevant intervals in its experiments, leaving open how stakeholders should elicit and validate such distributions in practice.
• Bridging validation-to-test constraint gaps. Several eICU models that satisfied constraints on validation fell short on test (precision of 0.44 instead of 0.5). Addressing this statistical variation — perhaps through more conservative selection or uncertainty-aware constraint handling — is a natural next question.
• Extending constraint-aware analysis beyond binary classification metrics. The authors position PV against cost-sensitive ROC extensions and partial AUROC variants; the boundary between what PV captures and what those other tools capture is not fully closed, leaving room for comparative study across tasks with different cost and class-balance regimes.

Target Audience

This paper is most useful to applied machine learning researchers and practitioners who deploy binary classifiers under operational constraints — especially clinical informatics teams building or evaluating deterioration-alert systems — as well as methodologists working on ROC analysis, cost-sensitive evaluation, and performance metrics for imbalanced or low-prevalence problems. Statisticians and ML theory researchers interested in the geometry of ROC space and 3D ROC-surface measures will also find the feasibility-region derivations relevant.

Authors’ abstract

The ROC curve is widely used to assess binary classifiers. Yet for some applications, such as alert systems for monitoring hospitalized patients, conventional ROC analysis cannot meet two key deployment needs: enforcing a constraint on precision to avoid false alarm fatigue and imposing an upper bound on the number of predicted positives to represent the capacity of hospital staff. The usual area under the curve metric also does not reflect asymmetric costs for false positives and false negatives. In this paper we address all three of these issues. First, we show how the subset of classifiers that meet precision and capacity constraints occupy a feasible region in ROC space. We establish the polygon-shaped geometry of this region. We then define the partial area of lesser classifiers, a performance metric that is monotonic with cost and only accounts for the feasible region. Averaging this area over a desired distribution for cost parameters results in the partial volume over the ROC surface, or partial VOROS. In experiments predicting mortality risk from vital sign history on several datasets, we show this cost-aware metric can outperform alternatives at ranking classifiers for in-hospital alerts.