Thresholds, Resource Allocation, and Fairness in Operations

Whoever writes the decision policy is the person shaping fairness. The decision policy is the rule that converts a score into an action under a constraint. Thresholds, ranking, quotas, budgets, review capacity, fallback and service supply decide how the real outcomes end up distributed.

One audited system shows that layer written out in full. A commercial care-management algorithm converted its score into action at two fixed percentile cut-offs. Patients above the 97th percentile were automatically enrolled in the high-risk care management program. Patients above the 55th percentile were referred to their primary care physician for consideration. Obermeyer and colleagues audited it in Science in 2019, over 6,079 Black and 43,539 White patients — 11,929 and 88,080 patient-years — at one large academic hospital between 2013 and 2015. At the 97th-percentile cut-off, Black patients had 26.3% more chronic illnesses than White patients: 4.8 distinct conditions against 3.8, P<0.001. In the counterfactual the authors ran, the two cut-offs never moved. Only what the score measured did. Their abstract reports the size of that: “Remedying this disparity would increase the percentage of Black patients receiving additional help from 17.7 to 46.5%.” Same thresholds, different target, and the 17.7% becomes 46.5%.

So a team evaluating the operation should simulate the queue and the resources behind it, not only confusion matrices. The measures that matter include selection rate, error type, wait time, service quality, and how much work the queue puts on reviewers. They include unmet need, repeat exposure, appeal, and the effect of capacity expansion or minimum guarantees.

Unmet need stays invisible in a confusion matrix. The cases that were never reviewed never became rows in it. The kidney queue shows how a case is removed before anyone scores it. A candidate only begins accruing qualifying waiting time once eGFR falls to 20 ml/min or below, even though OPTN policy allows registration on the list at any eGFR. A race coefficient that lifted an estimate above 20 therefore acted before the queue rather than inside it. Martha Pavlakis put the consequence plainly in 2023: “For the Black patient with an eGFR reflecting a value above 20, which if it was race neutral would be 20 or below, the opportunity for referral for transplant evaluation is lost.” That patient is not scored badly. That patient is not there.

Score, decision policy, capacity, human workflow, feedback: the score is the part everyone audits, and the other four decide who gets served.

The Allegheny Family Screening Tool makes the middle boxes concrete rather than schematic. The tool shows the call screener a risk score from 1 to 20. What selects a case is the policy wrapped around that number, together with the person applying it. A working paper from 2023, published in 2026 in the Journal of Policy Analysis and Management, describes the arrangement: “For referrals with the highest risk scores, the AFST defaults to a screen-in recommendation. In all cases, the call-screening supervisors remain the ultimate decision-maker, and may choose to override the default.”

The same paper measured what that decision policy did. It used difference-in-differences on Allegheny County referrals from 2010 to 2020. The tool cut screening-rate disparities between Black and White children by 19%. It cut within-score disparities by 2.7 percentage points, or 46% of the pre-existing gap.

Within-score is the phrase to stop on. Children carrying the same number were treated differently. So the movement was made in the default and the override — boxes two and four, where no retraining takes place.

Allegheny County's screening queue has been graded as an operation. An independent impact evaluation, run at Stanford by Goldhaber-Fiebert and Prince in 2019, measured four things: accuracy, workload, consistency across screeners, and disparities. That list is the argument of this lesson in miniature.

They compared 31,190 children in referrals before implementation (1 January 2015 – 31 July 2016) with 33,966 after (1 December 2016 – 31 May 2018). Consistency was checked across a subgroup of 11 call screeners who handled a substantial volume of referrals in both periods. They found roughly 24 more children screened in accurately each month, almost all of them in the white race group. Implementation also halted the pre-existing downward trend in the screen-in rate.

The conclusion they drew from a rising screen-in rate is a sentence about resources, not about the model: “Rather, it might imply that additional allocation of investigative resources is required to sustain improvements due to the AFST—a finding that is relevant for other systems considering implementing similar tools.” Screening more children in correctly is a bill arriving at the investigation unit.

Simulate the queue before touching the model. Volume, capacity and fallback shape who is served at least as much as the ranking does.

The Stanford evaluation is the field version of these five steps rather than an illustration of them. It did not report accuracy alone, but accuracy, workload, consistency across screeners and disparities. It fixed two dated intake windows and counted the children in each: 31,190 before, 33,966 after. It broke the accuracy gain out by group and found it almost all in the white race group. It ended on the investigative resources that sustaining the improvement would require.

Every one of those moves is available to a team that has a queue, a log and two dates. None of them requires retraining anything.

An allocation policy can be changed without retraining anything, and the redistribution is measurable.

On 5 January 2023, OPTN policy began requiring kidney programs to identify Black candidates disadvantaged by race-inclusive eGFR and backdate their waiting time. Programs had until 3 January 2024 to comply. A 2025 study in the Journal of the American Society of Nephrology counted what the change did: “Overall, 32% (14,419/44,912) of Black candidate listings received an eGFR modification of waiting time priority.” The median increase was 610 qualifying priority days (IQR 330–1,049). The adjusted hazard ratio for deceased-donor transplantation was 2.85 (95% CI 2.70–3.02).

No model was retrained. Queue position was.

A fixed threshold acts on a score. Top-k acts on capacity. A service guarantee reserves a floor for defined needs whatever the ranking says.

The top-k trade has been measured on both of its faces at once. Kleinberg and colleagues simulated bail release rules over New York City cases from 2008 to 2013, and reported the same policy in the two directions a fixed constraint allows. Hold the jailing rate constant, and the algorithmic rule cut crime by no less than 14.4% and up to 24.7%. Hold crime constant, and it cut jail rates by no less than 18.5% and up to 41.9%. Their abstract states it as one sentence: “one policy simulation shows crime reductions up to 24.7% with no change in jailing rates, or jailing rate reductions up to 41.9% with no increase in crime rates.” The gain is real, and it is not free-floating. Someone chooses which of those two numbers to take, and the choice is the policy.

The same 2017 study shows how far a human queue can sit from any consistent ranking. Stricter judges did not detain the highest-risk defendants first. Judges released 48.5% of the riskiest 1% of defendants. Those released failed to appear at a 56.3% rate and were re-arrested at a 62.7% rate.

The service guarantee is not a hand-wave either. It has a published mechanism and a name: FA*IR, a fair top-k ranking algorithm from 2017. The floor is defined and then enforced at every prefix of the list, not only at the cut: “Our ranked group fairness definition extends group fairness using the standard notion of protected groups and is based on ensuring that the proportion of protected candidates in every prefix of the top-k ranking remains statistically above or indistinguishable from a given minimum.” The paper reports that the resulting rankings show small distortion relative to the pure utility-maximising order. That is the empirical form of a claim worth holding onto. A floor can be combined with ranking. It does not have to replace it.

Fairness achieved at the model and lost at the queue is still lost. Wait, capacity and fallback are therefore part of what a reviewer has to see before believing a fairness claim.

Define when operational fairness and resource allocation requires the team to redesign, restrict, remedy, or retire the system.

The threshold itself has been studied directly. In 2017 Corbett-Davies and colleagues treated the decision as a constrained optimisation over decision thresholds, rather than as a property that a calibrated score either has or lacks. They quantified the gap on Broward County, Florida pretrial data. What that reformulation produces is not a warning but a shape you can look at: “We show that for several past definitions of fairness, the optimal algorithms that result require detaining defendants above race-specific risk thresholds. We further show that the optimal unconstrained algorithm requires applying a single, uniform threshold to all defendants.”

Race-specific thresholds, or one uniform threshold. Those are the two objects on the table, and each of them is a decision somebody has to sign. A fairness constraint on a threshold carries a cost. The cost falls somewhere specific. Which group pays it is a policy choice.

It is made whether or not anyone writes it down.