Application of predictive machine learning in pen & paper RPG game design

AI summary

Overview

• Research area: Machine learning applied to tabletop (pen & paper) role-playing game design, specifically ordinal regression for estimating monster difficulty levels in the Pathfinder Roleplaying Game Second Edition.
• Technical level: Intermediate (a master's thesis level treatment; it assumes familiarity with regression, ensemble methods, and classification metrics, but explains the domain concepts from scratch).
• Scope: A master's thesis that builds a dedicated dataset from Pathfinder 2e monster data, designs an evaluation procedure for chronologically ordered data, constructs a human-inspired baseline from the game's rulebooks, and surveys/compares regression and ordinal-regression algorithms for predicting a monster's level.

What This Paper Is About

Publishers of pen & paper RPGs currently have no automated way to determine how powerful—what "level"—a newly designed monster is; they rely on time-consuming manual playtesting sessions or on expert comparison against thousands of existing creatures. The thesis treats monster level as an ordered categorical target and asks whether machine learning can predict it accurately and efficiently from a monster's numerical statistics. Along the way it builds the dataset, a domain-grounded evaluation scheme, and a rulebook-derived baseline that mimics how a human designer would work.

Key Contributions

1. A novel dataset for ordinal regression. Monster statistics were collected from Pathfinder Second Edition rulebooks and textbooks, producing a tabular dataset (one row per monster, one column per attribute) with a level label that is an integer ranging between -1 and 27 in the source system. Monsters above 20th level, described as extremely rare, were aggregated into a single category labeled level 21.
2. An evaluation procedure for chronologically structured, imbalanced data. This includes a chronological data split and an expanding-window approach inspired by time-series evaluation, plus a metric set chosen to reflect the ordinal nature of the labels, temporal dependencies, and class imbalance.
3. A human-inspired baseline model derived from Pathfinder rulebook creature-building guidelines, so that ML algorithms can be compared against the approach publishers actually use.
4. An implementation and comparison of state-of-the-art ordinal regression models alongside classical continuous regression models with post-hoc rounding, addressing the level estimation problem.

Main Findings

• The task is framed as ordinal regression. Monster level is an ordered variable where the spacing between categories is not necessarily equal, which the paper argues makes dedicated ordinal methods a natural fit rather than plain classification or plain regression.
• Feature engineering is domain-driven. Eight core features were selected first (Strength, Dexterity, Constitution, Intelligence, Wisdom, Charisma, Armor Class, Hit Points), then extended with Perception, Fortitude, Reflex, and Will, and further with speed (land, fly, swim), number of immunities, number of spells per level from 1 to 9, spell Difficulty Class, spell attack, and max attack bonus plus average damage for melee and ranged attacks.
• Offensive power was converted into expected damage. For an attack written as "2d8 + 1d4 + 3", the expected damage is computed as the sum of each die's expected value plus the modifier, giving 14.5.
• Rounding of continuous predictions is a design choice, not a detail. The paper tests mathematical rounding (fixed threshold 0.5), a tuned global threshold, and tuned per-level thresholds—the last treated either as a hyperparameter or solved as a mathematical-programming graph shortest-paths problem.
• Class imbalance motivates the metric choices. Because ordinal regression datasets are often imbalanced, macro-averaged MAE and RMSE are adopted alongside RMSE and MAE; Somers' D is used for ordinal dependence, and accuracy plus accuracy@k (near-miss within k positions) capture classification-style performance.
• A gap in the literature is identified. Research intersecting computer science and RPGs focuses mostly on digital role-playing video games, storytelling, or technology-enhanced play; mechanical and game-balance problems such as monster level estimation in TTRPGs remain largely unaddressed.
• Experimental results are not reported in the available content. The retrieved text is truncated during the description of the single-best-threshold rounding strategy, so no benchmark tables, final model rankings, or error values are present to report.

Methodology in Plain English

The author scraped Pathfinder Second Edition monster entries, which arrive as deeply nested JSON with seven main fields and many subfields, and flattened them into a table of numerical attributes. Raw features were simplified and completed: skills that a monster cannot use were set to 0 (for example, a monster that cannot fly gets a fly speed of 0), missing values were imputed using domain knowledge, and features were normalized for models that need it, such as regularized regression. Immunities were collapsed into a single count because individual immunity types are rare and would create very sparse columns.

For prediction, two families were compared. The first used ordinary continuous regression models—LASSO, Ridge, LAD, and Huber linear models, Random Forest, LightGBM (tuned with the Tree Parzen Estimator in Optuna), SVM with linear and Gaussian kernels, and K-Nearest Neighbours—followed by a rounding step to turn real-valued outputs into integer levels. The second family used models designed for ordered targets, described in the literature review: the Simple Approach to Ordinal Classification (which splits into n−1 binary tasks), ORD, threshold-based linear models, Gaussian Process Ordinal Regression, and neural approaches such as CORAL and CORN that enforce rank consistency.

To imitate what a human designer does, the author read the rulebook's creature-building guidance, which assigns statistics on a scale of terrible, low, moderate, high, and extreme relative to a chosen level, with balance maintained by trading off between attributes. From these tables and the collected monsters, an algorithm was written that assigns a monster an archetype based on its characteristics and then predicts level from the closest monsters sharing that archetype—this is the baseline. Evaluation avoids leaking future information by using a chronological split and an expanding window, and it reports ordinal-aware metrics.

Why This Matters

• Research impact: The work describes a concrete, reusable ordinal regression benchmark drawn from a real, licensed, publicly available domain, and it contributes an evaluation recipe for datasets whose samples arrive in chronological order—something standard random splits handle poorly.
• Real-world applications:
  • Helping TTRPG publishers screen newly designed monsters before committing to expensive manual playtesting.
  • Giving independent and hobbyist designers a fast sanity check when creating custom creatures, instead of manually scanning thousands of published entries.
  • Supporting encounter-building tools and virtual tabletop platforms that need to estimate whether a proposed fight is fair for a given party.
  • Providing a template for other catalog-style balance problems where an ordered rating must be inferred from structured statistics.
• Industry relevance: The paper cites a TTRPG market valued at approximately $1.92 billion USD in 2023 with projections of $5.27 billion USD by 2033, and notes US and Canada sales rising 23% in 2019 and 31% in 2020, exceeding $100 million for the first time, with the market growing roughly seven times since 2013. It also notes that Hasbro, parent of Wizards of the Coast, has acknowledged incorporating AI systems into development of Dungeons & Dragons and Magic: The Gathering. Pathfinder itself is described as consistently ranking in the top three TTRPGs since release, holding first place for about two years and most often occupying second, with Paizo staff quoted calling Pathfinder 2e the best-selling, most successful thing Paizo has ever made.

Future Directions

• Report and stress-test the model comparison. The truncated content does not show final errors, so the natural next step is full reporting of RMSE, MAE, macro-averaged variants, Somers' D, accuracy, and accuracy@k across all models.
• Extend the feature set beyond the selected subset. The rulebook method the baseline imitates works best for physically strong, low-level monsters and does not describe adjustments for spellcasters or non-melee creatures; the same limitation may persist in the learned features.
• Improve handling of rare high levels. Levels above 20 were merged into a single category, and the imbalance visible in the accuracy figure suggests that better treatment of rare but valid levels—or of the ordering itself—remains open.
• Reuse the chronological/expanding-window procedure in other domains. Since publishers keep releasing new monsters over time, adapting to distribution shift as new books appear is an ongoing problem the paper's evaluation design only begins to address.

Target Audience

This paper suits machine learning practitioners interested in ordinal regression and non-standard tabular datasets, data scientists working in games or entertainment analytics, and TTRPG designers or publishers who want a quantitative alternative to manual playtesting. It is also useful for students looking for an end-to-end example of building a dataset, a domain-informed baseline, and an evaluation protocol for an ordered target on temporal data.

Authors’ abstract

In recent years, the pen and paper RPG market has experienced significant growth. As a result, companies are increasingly exploring the integration of AI technologies to enhance player experience and gain a competitive edge. One of the key challenges faced by publishers is designing new opponents and estimating their challenge level. Currently, there are no automated methods for determining a monster's level; the only approaches used are based on manual testing and expert evaluation. Although these manual methods can provide reasonably accurate estimates, they are time-consuming and resource-intensive. Level prediction can be approached using ordinal regression techniques. This thesis presents an overview and evaluation of state-of-the-art methods for this task. It also details the construction of a dedicated dataset for level estimation. Furthermore, a human-inspired model was developed to serve as a benchmark, allowing comparison between machine learning algorithms and the approach typically employed by pen and paper RPG publishers. In addition, a specialized evaluation procedure, grounded in domain knowledge, was designed to assess model performance and facilitate meaningful comparisons.