How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Use

AI summary

Overview

Research area: Artificial Intelligence, specifically large language model reasoning, game-theoretic decision-making, and agentic tool use in imperfect-information games.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (PPO, behavior cloning), counterfactual regret minimization (CFR+), Nash equilibrium, and tool-integrated reasoning, though the core narrative is still traceable for readers who only want the high-level conclusions.

Scope (one sentence): The paper systematically measures how far modern LLMs are from professional poker play, diagnoses three recurring reasoning flaws, tests whether RL fine-tuning can fix them, and then proposes ToolPoker, a tool-integrated reasoning framework that calls external poker solvers to deliver near-professional gameplay and reasoning.

What This Paper Is About

Poker is an incomplete-information game where professionals succeed not only by picking strong actions but by reasoning in a game-theoretic way, grounding decisions in equilibrium principles, equity, ranges, and pot odds. The paper asks how far LLMs actually are from that standard, evaluating both their gameplay outcomes (net chip gains against established algorithms) and the quality of their reasoning traces. Finding that LLMs fall short for three identifiable reasons, the authors move from trying to fix the model internally through behavior cloning plus reinforcement learning, to an alternative approach where the LLM uses external solvers as tools.

Key Contributions

1. A systematic study of LLMs in poker covering gameplay outcomes and reasoning traces, revealing three recurring flaws: heuristic reasoning, factual misunderstanding, and a "knowing–doing" gap where articulated reasoning and final actions diverge.

2. An initial two-stage RL attempt (BC-RIRL) combining behavior cloning on expert reasoning traces with regret-inspired reinforcement learning using step-level regret-guided rewards from a pre-trained CFR solver. This improves reasoning style and action–reasoning alignment but remains insufficient for accurate game-theoretic derivation or competitive play.

3. ToolPoker, described as the first tool-integrated reasoning (TIR) framework for imperfect-information games, which consolidates multiple poker solvers into a single unified tool interface so one query returns GTO actions plus supporting quantities such as equity and hand ranges.

4. Extensive experiments showing ToolPoker achieves state-of-the-art gameplay performance and reasoning traces that align closely with game-theoretic principles, plus ablations isolating the roles of behavior cloning and RL fine-tuning.

Main Findings

• Vanilla LLMs mostly lose to traditional algorithms. In Leduc Hold'em, Qwen2.5-3B averages −135.5 chips across NFSP, DQN, DMC and CFR+, and Qwen2.5-7B averages −73.0. In Limit Texas Hold'em, Qwen2.5-3B averages −205.8 and Qwen2.5-7B averages −121.6.

• CFR+ consistently beats every LLM tested. This includes strong closed-source models such as GPT-4o and o4-mini, which the authors attribute to CFR+ explicitly targeting Nash equilibrium strategies.

• Large and closed-source LLMs can exploit non-equilibrium opponents. Against NFSP, DQN and DMC in Leduc Hold'em, GPT-4.1-mini reaches +41.5, +60.5 and −22 chips, and GPT-4o reaches +34, +53 and −43 chips, whereas Qwen2.5-3B loses −143.5, −161 and −124 chips against the same baselines.

• Three reasoning flaws, demonstrated by case studies. (i) Heuristic reasoning: shallow pattern-based decisions instead of game-theoretic principles. (ii) Factual misunderstanding: GPT-4o judged (♠K, ♣10) as weak and preferred folding, while an equity calculator shows roughly 60% equity. (iii) Knowing–doing gap: Qwen2.5-3B correctly reasoned that (♣3, ♡10) is not strong and folding is optimal, yet proceeded to raise.

• A quantitative judge confirms the flaws persist. Using GPT-4.1-mini as judge on a 0–2 scale over 20 sampled traces per model, professional-style traces score 2 on all three metrics, while Qwen2.5-3B scores 0.53 HR, 0.18 FA and 1.53 AC in Leduc Hold'em, and even the strongest model, o4-mini, reaches only 1.80 / 1.56 / 1.85.

• Scaling helps but does not close the gap. Larger models (Qwen2.5-72B, o4-mini) improve all metrics, but significant factual-alignment and action-consistency gaps remain.

• BC-RIRL improves reasoning but not enough. It outperforms direct prompting and BC-SPRL in Leduc Hold'em (+17.0 chips vs. GPT-4.1-mini) but still trails CFR+ by 34.0 chips. Pure RIRL without the BC stage yields −64.5 chips vs. GPT-4.1-mini, indicating BC is a necessary foundation. RIRL raises HR and AC (1.93 HR and 1.90 AC in Leduc Hold'em, versus 1.80 / 1.85 for o4-mini) but improves FA only marginally (1.12, versus 0.87 for Qwen2.5-7B and 1.65 for o4-mini).

• ToolPoker delivers state-of-the-art gameplay. In Limit Texas Hold'em, ToolPoker gains +60.5, +63.0 and +61.5 chips against NFSP, DQN and DMC, while BC-RIRL gains −77.5, −82.5 and −80.5 against the same opponents. It only slightly trails equilibrium solvers: −3.0 chips against CFR+ in Leduc Hold'em and −5.0 against DeepCFR in Limit Texas Hold'em, which the authors attribute to occasional errors in tool calling.

• ToolPoker also produces near-professional reasoning. It achieves near-perfect scores across HR, FA and AC, outperforming all baselines, and yields substantially higher FA than BC-RIRL because its reasoning is grounded in solver-derived quantities.

• Ablations show both stages matter. ToolPoker/BC (removing BC) suffers lower HR and weaker gameplay — it can query the solver but does not internalize game-theoretic reasoning. ToolPoker/RL (removing RL) attains higher HR but performs poorly in gameplay and FA/AC, indicating superficial imitation. The full method is strongest overall.

Methodology in Plain English

The authors start by benchmarking a range of instruction-tuned LLMs (Qwen2.5-3B/7B/72B, Qwen3-8B, Llama3-8B, GPT-4.1-mini, GPT-4o, o4-mini) against four traditional opponents — NFSP, DQN, DMC and CFR+, with DeepCFR substituted for CFR+ in the more complex Limit Texas Hold'em — in the RLCard environment. Games are run with fixed random seeds and fixed player positions, then rerun with the same seeds but switched positions; each player starts with 100 chips, with a 1-chip small blind and 2-chip big blind, and performance is measured as net chip gain. To understand failures, they read reasoning traces and then score them quantitatively with an LLM-as-a-Judge setup using three metrics graded 0–2.

To test whether the flaws can be fixed inside the model, they build a small expert dataset: expert actions come from querying the CFR+ solver, reasoning traces are generated by an LLM using domain prompts about equity, pot odds and opponent ranges, and an automated pipeline checks action consistency against CFR+ and filters low-quality samples using the HR/FA/AC metrics. After filtering, roughly 5k reasoning-augmented samples remain, used for supervised fine-tuning (behavior cloning). They note that pure SFT at scale is impractical because prior work reports that mastering even Leduc Hold'em requires at least 400k action-only instances. A second stage, regret-inspired RL, converts cumulative regrets from a pre-trained CFR solver into normalized step-level rewards and fine-tunes with PPO.

Because internal policy training still falls short, the final approach gives the model tools. A structured prompt template has the model produce reasoning in <think> tags, issue a solver query in <tool> tags, receive results in <output> tags, and then emit its action in <answer> tags. All solver functionality is consolidated into a single unified interface so that one query returns the GTO action plus auxiliary statistics such as equity, pot odds and range distributions. Training is two-stage again: behavior cloning on the reasoning dataset programmatically augmented with standardized tool-call templates and execution outputs, then PPO with a composite reward combining an answer reward (against the CFR ground-truth action), a format reward, and a tool-execution reward, following the ReTool approach for multi-turn real-time tool use.

Why This Matters

The paper reframes the question from "can LLMs play poker well?" to "can LLMs reason like professionals, and what has to change for them to do so?" Its central claim is that internal policy training — even with dense, solver-derived regret rewards — cannot supply the precise game-theoretic derivations that professional play requires, whereas external tools can. That shifts attention from improving a model's internal policy toward designing reliable tool integration for strategic settings, with potential implications for the broader domains the authors mention, such as cybersecurity and negotiation.

Real-world applications suggested by the work:

• Strategic decision support under uncertainty, where a model must estimate hidden states and opponent behavior before committing to an action.
• Negotiation and other multi-agent settings with incomplete information, where reasoning traces need to be consistent with the final decision.
• Cybersecurity and other high-stakes domains the authors cite as motivations for studying reasoning under uncertainty.
• Agentic tool-use systems generally, since the paper's unified tool interface and multi-turn tool RL recipe are not specific to poker.

Industry relevance: The paper is co-authored by researchers from Amazon, Microsoft, Penn State, HKUST (GZ), and Tsinghua, and its training recipe (behavior cloning on tool-augmented traces, followed by RL with answer, format, and tool-execution rewards) is directly relevant to teams building tool-using agents that must make consequential decisions. The ethical statement notes that all experiments were run in simulated environments with no financial transactions or human participants.

Future Directions

• Extending TIR to more complex strategic settings, which the conclusion explicitly names as the direction the work paves the way for.
• Reducing tool-calling errors, since the residual gap to CFR+ (−3.0 chips in Leduc Hold'em and −5.0 against DeepCFR in Limit Texas Hold'em) is attributed to occasional failures in executing the solver.
• Closing the factual-misunderstanding gap that RL alone does not fix, since FA improved only marginally under RIRL (1.12 in Leduc Hold'em) compared with HR and AC.
• Making expert reasoning data cheaper at scale, given that the paper reports prior work needing at least 400k action-only instances for Leduc Hold'em and that adding reasoning traces multiplies both time and financial cost.

Target Audience

Researchers and practitioners working on LLM reasoning, reinforcement learning for language models, and agentic tool use, especially those interested in imperfect-information games or in benchmarks that evaluate reasoning traces rather than only final outcomes. It is also useful for readers who want a concrete, evidence-backed argument for why tool integration can outperform internal policy training on tasks requiring exact game-theoretic computations. The benchmark tables and judge metrics make it readable for those tracking LLM poker performance specifically.

Authors’ abstract

As Large Language Models (LLMs) are increasingly applied in high-stakes domains, their ability to reason strategically under uncertainty becomes critical. Poker provides a rigorous testbed, requiring not only strong actions but also principled, game-theoretic reasoning. In this paper, we conduct a systematic study of LLMs in multiple realistic poker tasks, evaluating both gameplay outcomes and reasoning traces. Our analysis reveals LLMs fail to compete against traditional algorithms and identifies three recurring flaws: reliance on heuristics, factual misunderstandings, and a "knowing-doing" gap where actions diverge from reasoning. An initial attempt with behavior cloning and step-level reinforcement learning improves reasoning style but remains insufficient for accurate game-theoretic play. Motivated by these limitations, we propose ToolPoker, a tool-integrated reasoning framework that combines external solvers for GTO-consistent actions with more precise professional-style explanations. Experiments demonstrate that ToolPoker achieves state-of-the-art gameplay while producing reasoning traces that closely reflect game-theoretic principles.