The Need for Benchmarks to Advance AI-Enabled Player Risk Detection in Gambling

AI summary

Overview

Research area: AI safety and ethics applied to gambling harm prevention — specifically the evaluation and governance of machine learning systems used to detect at-risk gamblers.

Technical level: Intermediate. The paper is conceptual rather than technical, but it assumes familiarity with machine learning evaluation concepts (sensitivity, precision, F1 score, out-of-sample testing, benchmark datasets).

Scope: A position paper proposing a conceptual benchmarking framework — structured, repeatable evaluation using standardized datasets, defined tasks, and agreed metrics — for AI-based player risk detection systems in gambling.

What This Paper Is About

Gambling operators, vendors, and regulators increasingly rely on AI and machine learning to identify players at risk of gambling harm and trigger interventions, yet there is no standardized way to evaluate whether these systems actually work. The authors argue that the field's critical next innovation is not a better model but a measurement framework, because without one it is impossible to gauge progress: new systems keep appearing while their comparative effectiveness remains unknown. The paper lays out the structural requirements, core dimensions, and practical considerations for building such benchmarks, aiming to start a discussion rather than deliver a finished benchmark.

Key Contributions

1. A diagnosis of the evaluation vacuum in player risk detection. The paper documents why current evidence is insufficient — fragmented academic studies using different datasets, parameters, and cleaning procedures, plus commercial claims that cannot be independently verified or compared.

2. A conceptual benchmarking suite with three core dimensions. The framework organizes benchmark datasets along Time (sessions, days, months), Engagement Level (frequency, regularity, historical duration of play), and Gambling Vertical (lottery, casino, sports betting, plus sub-categories).

3. Four illustrative benchmark dataset designs. Universal Benchmark Dataset, Lottery Players Benchmark Dataset, Highly Engaged Players Benchmark Dataset, and Early Detection Benchmark Dataset, each isolating or crossing dimensions.

4. Practical requirements for adoption. Specified data collection pathways (industry donation, regulator-mandated transfer, industry consortia), dataset documentation standards drawn from existing repositories, and an example metadata schema including benchmark tasks, target variables, and data fields.

Main Findings

• No formal benchmarks exist. Despite the success of benchmarking in other AI fields, the paper states that benchmarking in gambling studies remains nascent, with no formal benchmark datasets for player risk detection.

• Problem gambling prevalence varies enormously by context. An estimated 1.4% of adults globally meet criteria for problem gambling, but prevalence rises to approximately 16% among online casino or slots gamblers (Tran et al., 2024).

• The academic literature is substantial but fragmented. At least 40 academic studies exist alongside several literature reviews (Ghaharian et al., 2022; Delfabbro et al., 2023; Marionneau et al., 2025).

• Commercial models are numerous but opaque. Marionneau et al. (2025) identified 11 commercial models — six in-house systems from operators (e.g., Kindred, Veikkaus) and five from third-party vendors (e.g., Mindway, Openbet).

• Marker usage is inconsistent. The number of markers used in a given model ranges from fewer than 10 to over 100, and these markers are often unclear or inconsistently reported. Ghaharian et al. (2025b) mapped 65 "low-level" marker categories into five "high-level" categories: play, payment, engagement, RG tool use, and profile information.

• Vendor accuracy claims are not evaluable. Claims discussed at the International Association of Gaming Regulators annual conference (Percy, 2024) included models exceeding 80% accuracy (Sustainable Interaction), 87% (Mindway AI), and over 90% (Entain). The authors identify three problems: a high accuracy claim can be misleading (in a population where only 10% have a problem, a model could achieve 90% accuracy by identifying no one as at-risk); the figures are not comparable across differing risk definitions, prevalence, and out-of-sample testing approaches; and the claims are not independently reviewed.

• Generalizability is limited. Player risk profiles and model structures generalize poorly across brands (Ghaharian et al., 2024) and across time (Murch et al., 2024b), and likely across gambling type, jurisdiction, and regulatory context.

• Regulatory consequences are real. The UK Gambling Commission fined Corbett Bookmakers £0.7m in March 2025, with one identified issue being failure to identify a consumer who staked £23,674 in a 13-day period as someone who may be at risk.

• Time horizons in prior work vary widely. Studies have assessed future risk using as little as 7 days of data (Auer and Griffiths, 2024) or 30 days (Braverman and Shaffer, 2012), while others used 12-month datasets (Kairouz et al., 2023; Murch et al., 2023).

• Vertical-specific modeling may perform better. Prior work has suggested superior performance when vertical-specific models are developed (e.g., Kairouz et al., 2023).

• The closest thing to a de facto benchmark is dated. The bwin dataset from The Transparency Project — player account records from a European-based online sports betting provider, with a subset labeled by reason for account closure including "gambling problems" — enabled foundational studies (Adami et al., 2013; Braverman and Shaffer, 2012; Philander, 2013) but is limited by its age and single-operator origin.

• Data-sharing precedents exist but are limited. The UNLV Payments Research Collaborative published a peer-reviewed paper on a gambling payment transaction dataset with analytic subsets on a public repository (Ghaharian et al., 2023); Zendle and Newall (2024) released a synthetic dataset derived from banking transaction data of 424 UK-based survey respondents linked with their PGSI scores. The UK Gambling Commission worked with GambleAware to commission player account data across seven major operators (Forrest and McHale, 2022), though barriers limited third-party researcher access.

• Examples of structured data access. The Massachusetts Expanded Gaming Act requires casino licensees to provide anonymized customer tracking data to the Massachusetts Gaming Commission, which must make it available to qualified researchers (Andrews et al., 2025); the Netherlands Gambling Authority (Kansspelautoriteit, Ksa) mandates use of a Controle Data Bank (CDB).

• Example benchmark metadata. The paper's Table 1 illustrates a hypothetical "The Matrix Casino: Early Risk Dataset (Zion)" containing 105,677 sessions from 5,465 unique players during their first 7 days of activity, with a binary Risk_Flag target defined by a PGSI score of 5+ at 6 months, and two supervised classification tasks (7-day detection and 1-day detection).

• Benchmarking definitions from other domains. The paper cites Dueben et al. (2022) distinguishing scientific benchmarks (broad, long-term problems, self-reported performance via peer review) from competition benchmarks (narrower tasks, short-term community focus, e.g., Kaggle). It cites MNIST (created 1998, 70,000 size-normalized 28x28 pixel handwritten digit images), SVHN (created 2011, 600,000 labeled digits cropped from Street View images), MedMNIST for biomedical image classification, MMLU for LLM knowledge evaluation, and Vending-Bench for LLM-agent business simulation.

Methodology in Plain English

This is a conceptual and argumentative paper rather than an empirical study. The authors proceed in four stages. First, they review the landscape of machine learning for player risk detection, drawing on existing literature reviews to describe how many academic studies exist, how markers are categorized, and how many commercial models have been identified. Second, they lay out evaluation problems for different stakeholders — broad challenges such as sensitivity/precision trade-offs, the "black box" explainability problem, and missing contextual data, plus specific challenges for those procuring or commissioning models. Third, they introduce Performance Benchmarking as practiced in other AI domains, using MNIST, SVHN, MedMNIST, MMLU, and Vending-Bench as reference points, and describe the state of benchmarking, or its absence, in gambling. Fourth, they propose a benchmarking suite structure with defined dimensions and four illustrative dataset examples, then discuss the data collection, documentation, and repository requirements that would be needed for adoption. They explicitly frame the output as a discussion starter rather than a specified benchmark.

Why This Matters

Impact on research: The paper argues that without standardized datasets, tasks, and metrics, model performance cannot be compared across studies, progress cannot be tracked longitudinally, and good-faith operators cannot demonstrate to regulators or the public that their measures are high quality. It also notes that transparent reporting alone — even with established checklists like TRIPOD-AI, CONSORT-AI, and SPIRIT-AI — is only one step, because there are no methods to adjust performance metrics for like-for-like comparison across different contexts.

Real-world applications:

• Procurement decisions. Operators choosing between in-house builds and third-party vendors currently face competing marketing claims rather than comparable evidence; benchmarks would let them select on accuracy and efficacy rather than cost-effectiveness and ease of integration.
• Regulatory oversight. Benchmarks could serve as regulatory infrastructure, giving regulators a transparent framework for evaluating systems whose deployment is increasingly mandated.
• Player protection design. Better evaluation supports decisions about where to place the sensitivity/precision balance, so interventions reach those who need them without over-triggering and undermining trust in Responsible Gambling programs.
• Early detection. The framework specifies an Early Detection Benchmark Dataset using only a player's initial days of activity (e.g., days 1–7) to predict a negative outcome much further in the future (e.g., a risk flag at 6 months), directly targeting timely intervention.

Industry relevance: The paper points to a paradox: operators can claim adequate Responsible Gambling measures without revealing system specifics, which makes it hard for good actors to distinguish themselves and creates risk of reputational damage and regulatory action, as in the Corbett Bookmakers case. It also notes that AI ethics principles are increasingly being translated into regulation, citing the EU AI Act approved in May 2024, and that interested parties include researchers, operators, vendors, and regulators.

Future Directions

• Who builds and hosts the suite? The paper notes that it is unlikely global gambling stakeholders will collectively cooperate on a single comprehensive dataset, and that curating one would be significantly complex and costly; the practical alternative it sketches is gradual curation and procurement from various stakeholders over time.

• How should harm be defined as a target variable? The authors state that a benchmarking solution must tackle the definition of the target outcome, since the concept of harm in gambling is ambiguous, contentious, and inherently multifaceted. They acknowledge that validated self-report measures such as PGSI are not always available, so tasks may need to be defined around other harm proxies (discussed in the paper's Section 5.3, which is not fully included in the provided text).

• How can data accessibility be solved? A primary barrier identified is that player data is commercially sensitive and proprietary. Open questions include whether regulator-coordinated requests, legislative mandates, industry consortia, or synthetic data generation can provide sufficient breadth and generalizability without operator cooperation.

• How can quality be assured? The paper cites a lack of quality review of datasets at existing repositories and proposes requiring peer-reviewed introductory papers to accompany datasets, similar to Data in Brief. Whether repositories can enforce such review remains open.

• How are composite benchmarks designed? Real-world datasets will likely cross multiple dimensions simultaneously; the paper gives the examples of detecting risky player sessions on slot products (Gambling Vertical plus Time) and assessing algorithms on more highly engaged, high-value casino players (Gambling Vertical plus Engagement Level), but leaves the operational design of these to future work.

Target Audience

This paper is most useful to gambling regulators and policy staff designing oversight of AI-based player protection; operators and procurement managers evaluating risk detection vendors or in-house models; third-party vendors seeking credibility through transparent comparison; responsible gambling and public health researchers who need standardized evaluation infrastructure; and AI ethics researchers interested in how benchmarking can function as regulatory infrastructure in a commercially sensitive, high-consequence domain. It is written to be accessible to informed non-specialists but uses machine learning evaluation terminology without defining it.

Authors’ abstract

Artificial intelligence-based systems for player risk detection have become central to harm prevention efforts in the gambling industry. However, growing concerns around transparency and effectiveness have highlighted the absence of standardized methods for evaluating the quality and impact of these tools. This makes it impossible to gauge true progress; even as new systems are developed, their comparative effectiveness remains unknown. We argue the critical next innovation is developing a framework to measure these systems. This paper proposes a conceptual benchmarking framework to support the systematic evaluation of player risk detection systems. Benchmarking, in this context, refers to the structured and repeatable assessment of artificial intelligence models using standardized datasets, clearly defined tasks, and agreed-upon performance metrics. The goal is to enable objective, comparable, and longitudinal evaluation of player risk detection systems. We present a domain-specific framework for benchmarking that addresses the unique challenges of player risk detection in gambling and supports key stakeholders, including researchers, operators, vendors, and regulators. By enhancing transparency and improving system effectiveness, this framework aims to advance innovation and promote responsible artificial intelligence adoption in gambling harm prevention.