Actions Have Consequences: Detecting Outcome Performativity using Intervention Testing

AI summary

Overview

Research area: Machine learning performativity — specifically Outcome Performativity, where a model's predictions causally change the outcomes they are meant to predict — combined with statistical hypothesis testing and sample complexity analysis.

Technical level: Intermediate. The core idea (randomised intervention testing) is intuitive, but the paper derives formal sample complexity bounds that require comfort with Bernoulli outcome models, Chi-Squared tests and asymptotic analysis.

Scope: The paper formalises an offline, intervention-based test for detecting Outcome Performativity in binary classification settings, derives the minimum number of instances needed to detect it under three assumption classes, and validates these bounds on six binary datasets plus the Open Bandits Dataset. (Note: the provided paper content is truncated mid-sentence in the Discussion section, so the limitations list and any later material are not fully reported here.)

What This Paper Is About

Many prediction settings — credit assignment, palliative care, recommender systems — may be Outcome Performative: the prediction itself causally changes the outcome it predicts. A loan denial can make a firm more likely to go bankrupt; an assignment to palliative instead of curative care can affect whether a patient survives. The paper's goal is to detect whether such a setting is Outcome Performative before a model is trained and deployed, by randomly intervening on the predictions (as in an A/B test or randomised controlled trial) and checking whether the resulting outcome distributions differ significantly.

Key Contributions

1. A formal detection method (OPAB). Outcome Performativity A/B Detection is introduced as an offline procedure: randomly assign predictions in {0, 1} drawn from a Bernoulli(0.5) distribution to instances, observe the outcomes, and apply a hypothesis test to the two outcome distributions (Eq. 7). If the dissimilarity test is significant at threshold δ, Outcome Performativity is detected. Pseudocode is given in Algorithm 1.

2. Sample complexity bounds for three assumption classes. The paper derives the estimated minimum number of interventions per prediction group, N, for Simple Outcome Performativity (Eq. 9), Model-based Outcome Performativity (Eq. 12, following Mendler-Dünner et al.), and Misclassification-based Outcome Performativity (Eq. 14, following Adam et al.).

3. The concept of "regions of indistinguishability." Formally defined (Eq. 10) as the set of parametrisations of an assumption class where the allotted interventions per group (N̄) are less than the minimum required by the complexity bound — i.e., settings where detection is practically infeasible with the available budget.

4. Empirical validation and a real-data case study. The bounds are validated on the breast cancer, diabetes, adult census, kickstarter, titanic and loan binary classification datasets imputed with simulated performativity, plus a case study on the Open Bandits Dataset (OBD) collected on the fashion e-commerce platform ZOZOTOWN.

Main Findings

• Detection is achievable in many configurations but not all. OPAB detects Outcome Performativity across numerous parameter settings, but "regions of indistinguishability" appear consistently — regions where the allotted interventions are insufficient.

• Simple Outcome Performativity: as |α0 − α1| → 1.0, fewer instances are needed, and with N̄ = 10 detection is reliable when the difference is large. As α0 ≈ α1, N → ∞ and even N̄ = 10,000 cannot detect it. Fig. 2 reported detection rates for N̄ = {10, 100, 1000, 10000}, with each cell based on 25 repeated runs.

• Bound generalises across tests. Although Eq. 9 assumes a Chi-Squared test, the paper reports that the estimated N also applies when using Fisher's Exact Test and the Mann-Whitney U Test (Section D.3).

• Model-based Outcome Performativity behaves like the Simple case in form (Eq. 12), with N → 0.0 as |β0 − β1| → ∞ and N → ∞ as |β0 − β1| → 0.0 — but the rate depends on the parametrisation of fθ. Experiments sampled 10% of instances without replacement. Class imbalance matters: on the breast cancer dataset (imbalanced in favour of Y = 1), as βc → 1.0, N increases as the outcome distributions rapidly converge; the effect was seen less intensely on diabetes, which is slightly imbalanced in favour of Y = 0.

• Misclassification-based Outcome Performativity is the easiest to detect. Regions of indistinguishability were less common (Fig. 5, black regions), because only one parameter combination (γ0 = 1.0, γ1 = 1.0) is truly non-Outcome Performative, versus infinite such combinations for Simple (α0 = α1) and Model-based (β0 = β1). As γ0 and γ1 → 1.0, N → ∞, so weaker performativity is harder to detect.

• Case study on the Open Bandits Dataset. Applying OP-AB to all instances found Outcome Performativity in the men sub-dataset (p = 0.014) but not in women (p = 0.87) — meaning the placement order of recommended items was relevant for men but not for women. Across samples of N = {10, 10², 10³, 10⁴, 10⁵} over 1000 replicates, detection rates on women were 0.0, 0.001, 0.042, 0.042 and 0.016 (low false positive rate of about [0.0, 0.05]); on men they were 0.0, 0.0, 0.05, 0.094 and 0.523. The paper describes this as roughly 50% accurate at the largest N = 10⁵.

• Effect size and imbalance limit detectability in OBD. The low Phi coefficient (Φ = 0.004) indicates a negligible effect size, and the datasets are heavily imbalanced — only about 0.05% of item recommendations resulted in a click in the paper's setup. On balanced variants (random downsampling of the majority class, 1000 replicates, N = {10, 50, 100, 500, 1000}), detection rates were 0.046, 0.059, 0.06, 0.051 and 0.032 for women, and 0.044, 0.068, 0.074, 0.159 and 0.234 for men. The paper reports OP-AB was four times more effective at detecting Outcome Performativity when N = 10³ on the balanced men dataset.

• Setting used throughout. Unless stated otherwise, results are averaged over 100 replicates with a pseudorandom number generator for reproducibility, using the Chi-Squared Test with δ = 0.05 (critical χ² value with one degree of freedom is 3.841).

Methodology in Plain English

The authors start from a problem: if you look at the predictions made by an existing expert or model, those predictions are themselves derived from the instance features, so you cannot tell whether the prediction is causing the outcome or merely correlated with it. Their fix is to break that dependency by doing rather than observing — the do-intervention idea from causal inference.

Concretely, OPAB takes a set of T instances and flips a fair coin for each one, assigning a random prediction of 0 or 1 (equivalent to y ∼ Bernoulli(0.5)). Each instance is then observed under its assigned prediction, producing two groups — the A and B groups of an A/B test. The authors build frequency tables of the outcomes in each group and apply a statistical test (Chi-Squared or Fisher's Exact). If the two outcome distributions differ significantly at threshold δ, Outcome Performativity is detected.

To say how many instances that takes, the authors assume the performative effect is uniform across the feature space, which collapses the conditional test into a simple comparison of P(Y | do(Ŷ = 0)) and P(Y | do(Ŷ = 1)). They then model the underlying "Nature" p*(x, ŷ) in three ways — Simple (outcome depends only on the prediction), Model-based (an instance is shifted in feature space along a linear model's weights by a strength β), and Misclassification-based (performativity only bites when a prediction mismatches a domain-defined desirable outcome) — and derive closed-form expressions for the required sample size per group from Chi-Squared testing theory. They then check these formulas empirically by taking six non-performative binary datasets and artificially imputing performativity into them, subjecting the resulting data to OPAB and mapping detection rates against sample size and effect size. Finally they check whether a real dataset has the two properties needed for OPAB — recorded predictions and random assignment of those predictions — which the Open Bandits Dataset satisfies for a subset where item placement was randomly determined.

Why This Matters

Impact on research: The paper claims to be the first to explicitly derive sample complexity bounds for detecting Outcome Performativity, and moves the discussion from purely theoretical or semi-synthetic identifiability toward a practical, offline test. It reframes Outcome Performativity detection as a design question about experimental budget rather than only a modelling question.

Real-world applications:

• Palliative and curative care decisions — where assigning a care pathway can influence patient survival, and large-scale intervention testing would be ethically fraught.
• Credit assignment — where the decision to grant or deny a loan can change whether the borrower defaults or becomes insolvent.
• Recommender systems — the Open Bandits case study shows item placement affecting clicks for men on ZOZOTOWN.
• Any high-stakes deployment where performative feedback loops can silently inflate the False Positive Rate over time if practitioners unknowingly train on data their own predictions shaped.

Industry relevance: Because OPAB is offline, it can be run during data labelling and before a model is trained or deployed, which the authors argue can be more cost-effective and prevent unwanted performative effects arising after deployment. The sample complexity formulas give practitioners a way to estimate the minimum sample size needed, which matters directly when interventions are costly, scarce or ethically ambiguous. The authors also note the flip side: if effects are subtle and interventions are expensive or unethical, it may be inappropriate to deploy ML models in that domain unless non-interventional detection methods can be used — and those tend to be less sample efficient.

Future Directions

1. Extending beyond the uniform-effect assumption. The paper assumes Outcome Performative effects are uniform across the feature space to reduce Eq. 6 to Eq. 7, and states that the limitations of this assumption are elaborated in Section 6 (the Discussion text is truncated in the provided content). Relaxing this would require approaches such as T-learner or Double ML, which the paper notes are sensitive to intervention imbalances, hyper-parameter and model choices.

2. Addressing verification issues. The final visible sentence is "we did not consider the verification late", suggesting a limitation or follow-up direction that is cut off in the provided content and not reported here.

3. Finding or constructing datasets with explicit Outcome Performativity. The authors state that to their knowledge no dataset explicitly studies Outcome Performativity, and that detection is only inferable when predictions are recorded and randomly assigned — a strong constraint that currently limits real-world validation to datasets like OBD.

4. Non-interventional detection for infeasible regions. For settings inside the regions of indistinguishability where interventions are too costly or unethical, the open question is whether alternative, non-interventional methods can detect Outcome Performativity despite being less sample efficient.

Target Audience

Researchers working on performative prediction, causal inference in machine learning, and distribution shift; practitioners in high-stakes domains such as healthcare, credit and recommender systems who need to know whether their prediction pipeline changes the outcomes it predicts; and methodologists interested in sample complexity bounds and experimental design for hypothesis testing under causal feedback. Readers should be comfortable with binary classification notation, Bernoulli models and classical statistical tests, since the derivations (in Sections B and the Appendix, which are not fully included in the provided content) carry much of the technical weight.

Authors’ abstract

In many domains such as Palliative Care, Credit Assignment and Recommender Systems, predictions may causally influence the outcomes they predict. This phenomena is known as Outcome Performativity. This paper formalises an approach for detecting Outcome Performativity using prediction intervention called Outcome Performativity A/B Detection (OPAB). OPAB enables the detection of Outcome Performativity by assessing the dissimilarity in outcome distributions produced by different predictions groups (interventions). If that dissimilarity is significant, Outcome Performativity is detected. We derive sample complexity bounds for OPAB under various Outcome Performative assumption classes which we empirically validate. Results show that detecting Outcome Performativity using OPAB is achievable in numerous cases. Results also show the presence of regions of indistinguishability which describe settings where the allotted number of interventions are insufficient for detecting Outcome Performativity. The results of which have broader practical implications for the detectability of Outcome Performativity in settings where samples are scarce, cost-prohibitive or potentially unethical to obtain. The paper concludes with a case study on the efficacy of OPAB on the Open Bandits dataset, and provides directions for future work.