Capacity Constraints Make Admissions Processes Less Predictable

AI summary

Overview

Research area: Machine learning theory and social prediction, specifically the reliability of ML models used to predict admissions outcomes (college, high school, hiring) when decisions are capacity-constrained.

Technical level: Advanced. The paper combines formal choice-function theory from economics (substitutability, total orders, queues) with binary classification theory and an empirical ML study on real admissions data.

Scope: This paper defines two new properties of admissions decision rules — instability and variability — proves how they limit what standard machine learning can represent, and illustrates the theory on individual-level data from the New York City high school matching system.

What This Paper Is About

Machine learning models are routinely trained on historical admissions decisions to predict who will be admitted, and these predictions are used by both decision makers (recruiters, admissions officers) and applicants. The paper argues that this framing is fundamentally mismatched to the problem: admissions are capacity-constrained, so whether one applicant is admitted depends on who else applies, and that dependence is not captured by models that score each applicant independently.

The goal is to explain why admissions prediction is hard even in otherwise ideal settings, by tying the difficulty to formal properties of the schools' decision rules, and to demonstrate the effect using real data from the NYC high school match.

Key Contributions

1. Formal properties of admissions decision rules. The authors introduce instability (the number of existing decisions that can change when a single new applicant is added) and variability (the number of distinct currently accepted students whose decisions can change), characterizing how sensitive a decision process is to applicant-pool composition.

2. A bridge between choice-function theory and machine learning. They prove results connecting instability, variability, substitutability, and total orderings to what forms of ML models can faithfully represent admissions decisions.

3. An empirical illustration on the NYC high school matching system. Using admissions and applicant data, they characterize the instability and variability of the actual choice functions used by NYC programs and show that ML performance degrades as the applicant pool shifts away from the training distribution.

4. Simulator-based counterfactual evaluation. They use a rule-based simulator to generate admissions outcomes under alternative choice functions and applicant pools, isolating the effect of pool composition from other causes of prediction difficulty (including synthetic 0-unstable and 5-unstable methods), and release their code (with synthetic data results in the appendix).

Main Findings

• Capacity constraints forbid perfectly stable decisions. A q-acceptant choice function (one that accepts min{q, |X|} applicants) cannot be 0-unstable: as the pool grows, a school necessarily becomes more selective, so existing decisions must change when a new applicant is introduced.

• Substitutability is exactly equivalent to 1-instability. A q-acceptant choice function is exactly 1-unstable if and only if it is substitutable (removing other applicants cannot hurt an already-accepted applicant). Violations of substitutability must produce larger changes in decisions.

• Instability can be far larger than one. q-acceptant choice functions can be tightly d-unstable for every 1 ≤ d ≤ 2q. The paper's examples include complementary-group (team formation) mechanisms: a music department admitting string trios is 5-unstable and admitting quartets is 7-unstable. Lemma A.5 shows complementary groups of size n define tightly 2n−1 unstable functions, and Section A.6 constructs instability of 2n where new applicants change decisions without being accepted themselves.

• Standard independent ML models can represent almost nothing. A model of the independent form f(x) = 1{s(x) ≥ t} can only represent 0-unstable choice functions. A capacity-aware model that ranks applicants and applies a cohort-dependent threshold t_q(X) can represent 1-variable, 1-unstable choice functions. No such model can represent a choice function with variability or instability greater than one.

• Variability is bounded by the number of queues. For a q-acceptant, 1-unstable choice function expressible as a sequential composition of n choice functions each characterized by a total order, variability m satisfies 1 ≤ m ≤ n, and m = 1 if and only if the function is characterized by a single total order (n = 1). The striking corollary: a choice function can be faithfully represented by an ML model if and only if it is characterized by a total ordering over applicants.

• NYC high school programs are 1-unstable with variability equal to their queue count. All four analyzed function classes are 1-unstable, with variability equal to their number of queues: Screened/Open programs are 1-variable, Screened/Open with DIA are 2-variable, Ed. Opt. programs are 3-variable, and Ed. Opt with DIA are 6-variable.

• Real ML performance degrades under applicant-pool shift. Training on one year's pool and testing on pools that increasingly contain the other year's applicants, machine learning performance degrades as the applicant pool differs more from the training data, and there are larger performance drops for schools using decision rules that are more unstable and variable.

• The simulator is accurate. The authors' rule-based simulator, implementing publicly available NYC policies, reproduces actual admissions outcomes with 98.79% accuracy on 2022 pools and 91.47% accuracy on 2021 pools. Accuracy is under 100% because some criteria (e.g., children of teachers, siblings of current students) are not in the data.

Methodology in Plain English

The authors start from an economic concept — the choice function — which maps any set of applicants to the subset that gets accepted. This lets them describe admissions decisions as depending on the whole applicant pool rather than on each applicant alone. They build on this by defining two measurements: instability, which counts how many existing accept/reject decisions can flip when one new applicant joins the pool (using a "choice distance" between the accepted sets for two pools), and variability, which counts how many different currently accepted students could be displaced by adding a single new applicant.

They then prove theorems linking these measurements to well-known properties (substitutability, being expressible as a total ranking, or as a sequence of ranked queues) and to what a machine learning model can and cannot represent. A key empirical step is that the authors do not simply use the historical admissions outcomes; they use a simulator of the actual NYC decision rules so they can construct counterfactual outcomes — the same programs under different admissions methods, the same method under slightly altered applicant pools, and synthetic methods (including 0-unstable and 5-unstable ones) that no real program uses.

In the empirical study, they train logistic regression models with L2 regularization separately for each program and admissions function, tailoring the feature space so that training performance is near-perfect. Predictions come from ranking applicants by the model's score and accepting the top q, which the authors find improves average performance relative to a fixed 0.5 threshold — consistent with their Proposition 1. They then create mixture pools combining a γ fraction of 2021 applicants with 1−γ of 2022 applicants for γ ∈ {0, .1, .2, ..., 1}, and measure how accuracy changes as the test pool moves away from the training pool. Every combination of admissions method with each program is generated (six in total: three admissions methods, with or without 50% of seats reserved for DIA).

The exact numerical degradation figures for this experiment are not included in the provided excerpt of the paper, which is truncated mid-sentence at the description of the mixture pools.

Why This Matters

The paper identifies a mechanism — cohort dependence induced by capacity constraints, producing concept drift in y|x even when the decision rule is unchanged — that explains why admissions predictions fail, beyond commonly cited causes like bias, poor data, or distribution shift in features. It suggests that developing robust predictors in these settings requires modeling how decisions depend on entire applicant pools, not just individual applicants, and it reconnects ML practice to classic ideas from choice theory (substitutability, stable matching).

Real-world applications:

• Applicant-facing admissions chance calculators. NYC's Department of Education deployed a model in the 2024-2025 application year to help students assess odds of admission to high schools. That tool sidesteps these problems because it directly simulates the known choice functions rather than learning from data — an approach the paper treats as one path forward, but one that may not be feasible where choice functions are unknown.

• Holistic college admissions and hiring. When programs use essays, auditions, or other non-numeric criteria, the choice function is not fully known or simulable, so a machine learning component may be necessary — precisely the setting where the paper's challenges bite.

• Recruiters and resume filters / candidate ranking. Employers ranking applicants against a fixed number of openings face the same capacity constraint and the same cohort dependence.

• Bias auditing and litigation support. Statistical models of admission probability have played an important role in auditing, for example in Students for Fair Admissions v. Harvard; the paper's results raise questions about the reliability of individual-level admission probability estimates used in such analyses.

Industry relevance: Vendors and teams that sell or deploy predictive scoring for admissions, hiring, and other selection pipelines should be aware that independent per-candidate scoring models cannot represent capacity-constrained decisions with instability or variability greater than one, and that accuracy will erode as applicant pools change — even if the decision rule never changes.

Future Directions

• Model applicant pools, not just applicants. The results suggest robust predictors require explicitly modeling how decisions depend on the whole applicant pool, motivating new model classes and training objectives beyond independent classification.

• Determine when simulation is feasible. The NYC case shows that simulating known choice functions avoids the problem; open questions remain about hybrid approaches for settings where the rules are only partly known (e.g., holistic admissions, auditions, essays).

• Extend the theory beyond the analyzed cases. The paper notes that the general definition of variability (beyond 1-unstable and q-acceptant functions) is deferred to Section A.8, and that Theorem A.11 extends the queue-composition result to optimization-based admissions such as linear assignment problems — further extensions are left open.

• Understand implications for applicants, not just models. The authors frame their work as raising questions about the reliability of predicting individual admissions probabilities, which bears directly on how guidance tools and interventions for applicants should be designed and communicated.

Target Audience

Researchers working at the intersection of machine learning and high-stakes social decisions; algorithmic fairness and AI ethics scholars; economists and mechanism-design researchers interested in choice functions, substitutability, and matching markets; education policy analysts and school-district data teams building or evaluating admissions prediction tools; and practitioners in college admissions, recruiting, and hiring who deploy or rely on predictive scoring systems.

Authors’ abstract

Machine learning models are often used to make predictions about admissions process outcomes, such as for colleges or jobs. However, such decision processes differ substantially from the conventional machine learning paradigm. Because admissions decisions are capacity-constrained, whether a student is admitted depends on the other applicants who apply. We show how this dependence affects predictive performance even in otherwise ideal settings. Theoretically, we introduce two concepts that characterize the relationship between admission function properties, machine learning representation, and generalization to applicant pool distribution shifts: instability, which measures how many existing decisions can change when a single new applicant is introduced; and variability, which measures the number of unique students whose decisions can change. Empirically, we illustrate our theory on individual-level admissions data from the New York City high school matching system, showing that machine learning performance degrades as the applicant pool increasingly differs from the training data. Furthermore, there are larger performance drops for schools using decision rules that are more unstable and variable. Our work raises questions about the reliability of predicting individual admissions probabilities.