PUBG Ally: A Conversational Embodied Agent as an AI Teammate

AI summary

Overview

Research area: Artificial intelligence / embodied conversational agents, with a focus on real-time game AI and on-device language-model deployment.

Technical level: Advanced.

Scope: The paper describes the design, data collection, training, evaluation, and live-service deployment of PUBG Ally, a voice-enabled language-model agent that plays as a duo teammate with human players in PUBG: BATTLEGROUNDS.

What This Paper Is About

Building an AI that can play a fast-paced game alongside a human requires two capabilities that are hard to combine: reacting to a constantly changing world under strict latency limits, and holding a natural spoken conversation with a partner. PUBG Ally targets both at once, so its speech and its in-game actions stay consistent with each other. The paper's goal is to explain how the team built, trained, safety-hardened, and shipped such a teammate into a live commercial battle-royale service.

Key Contributions

1. An architecture for real-time gameplay plus voice interaction. Ally combines language-model reasoning, speech input and output, and autonomous gameplay through a bounded tool interface and a two-layer control hierarchy modeled on the System 1 / System 2 distinction.
2. Large-scale interaction data and training from real-player gameplay. The team collected nearly 39k gameplay sessions with real players and trained an on-device model using teacher demonstrations plus teacher-corrected student rollouts.
3. Player-centered evaluation of teammate quality. They built an evaluation framework grounded in player preferences and actual matches, covering conversational quality, gameplay behavior, and cooperation, and revised criteria where internal evaluations disagreed with players.
4. Contextual safety and production engineering for live deployment. They developed safety mechanisms that keep ordinary in-game combat talk playable while handling harmful speech and unsafe information entering persistent memory, and described the work needed to run language and speech models on-device under real-time constraints, deployed as a multilingual live-service beta.

Main Findings

• The agent is a co-playable character, not a solo bot. Ally joins a live match as a duo partner, coordinating a drop location, following voice commands while looting, calling out enemies, supporting the player in combat, and reviving a downed player (for example by deploying smoke for cover, moving to the player, and attempting a revive).
• A bounded tool interface replaces full-state input. Ally uses 16 callable tools grouped into six functional groups: match observation (7), knowledge and memory (2), action execution (2), player communication (3), agent loop control (1), and safety control (1).
• Two-layer control keeps the language model off the latency-critical path. System 2 (the language model agent) is invoked by events and issues high-level intents; System 1 (a behavior tree) is evaluated every tick and executes movement, combat, and recovery.
• Context is deliberately bounded. Ally operates with a context budget of roughly 5,000 tokens and uses compaction: match facts from observation tools are not carried forward, while a short plan retains decisions, pending tasks, promises, and unresolved events. A cache-friendly prompt layout orders system prompt, tool definitions, event history, carried-over plan, and current turn inputs.
• Reactivity is tunable without retraining. Per-event reactivity priors independently specify how strongly an event favors speech and action (require, recommend, optional, do-not), and controlled proactivity is bounded through event cooldowns, priors, and event phrasing.
• Data came from real matches at scale. Across 28 collection days at a rented PC bang in Korea, 1,046 participants completed 38,956 gameplay sessions; each session lasted 14.1 minutes and contained 59 trajectories on average.
• Training used teacher demonstrations plus corrections. After two weeks of data collection with the 31B teacher model, successive SLM versions played with humans for two weeks so rollouts reflected student-induced situations; the teacher generated corrections inspired by DAgger. The initial corpus contained 464K examples (420K teacher rollouts and 44K synthetic and safety examples), and correction sets contained 313K examples in total, with every student starting from the same pretrained checkpoint.
• On-device execution was faster than cloud. In measurements taken during the gameplay sessions, a single spoken exchange completed in approximately 1.6s on-device versus 3.4s with the cloud configuration.
• Live-service survey results were positive. The survey reached players in 141 countries. Among respondents whose play with Ally was confirmed in game records, positive responses exceeded negative responses by 25.1 percentage points on whether they would recommend Ally. When asked how they framed Ally, 18.5% selected "teammate" and 31.5% selected companion framings, together 50.0% of respondents.
• Notable first and external influence. To the authors' knowledge, Ally is the first conversational embodied teammate in a commercial live battle-royale game that reasons, acts autonomously, and coordinates through voice with its language and speech models running on-device. They note that Ludi 0.1 (Ludo Robotics, 2026) draws on Ally's agentic design principles for physical robotics.

Methodology in Plain English

The team treated the game as a shared environment in which a language model cannot run fast enough to control every action, so they split the job. A language model acts as the slow, deliberate layer: it is woken by selected events such as player speech or consequential game changes, looks at the world only through a fixed set of tools, decides what to say, and commits to a high-level action. A separate behavior tree runs every game tick and turns those commitments into movement, shooting, and revives.

To keep the conversation trustworthy, the tools expose only information the agent is allowed to see, the agent must check whether an action is available before dispatching it, and anything learned from observation is discarded at the end of each loop so stale facts do not leak into later decisions. Only a short plan survives across loops.

Because a teammate's behavior and a player's behavior continuously shape each other, the team could not train from a static corpus. They built a playtest environment, had real players play full matches with Ally, and recorded player speech, game events, requested observations, tool results, agent speech, and executed actions. Training started with a 31B teacher whose prompt was optimized using GEPA on human-annotated held-out trajectories, then moved to student models that themselves played with humans; a teacher then rewrote selected student interactions into corrected trajectories that were added to the growing training set, with synthetic examples filling gaps in rare game states, infrequent actions, and uncommon player requests. They also trained an intermediate 8B teacher and distilled a deployable 2B student.

Evaluation could not rely on combat performance alone, so the team used player preferences, survey responses, and interaction records, and specifically studied cases where internal evaluations disagreed with what players preferred, then revised their criteria. For deployment, they compressed models, compacted context, applied targeted safety training, added runtime guardrails, and redacted unsafe memory.

Why This Matters

This is a full-stack account of shipping a conversational embodied agent into a live commercial game, rather than a benchmark-only result. It shows how a language model can be kept out of the real-time control loop while still steering behavior, and it treats player perception of teammate quality as the primary evaluation signal. The reported on-device latency (approximately 1.6s per spoken exchange versus 3.4s in the cloud configuration) and the on-device memory constraint (consumer GPUs with at least 8 GB of VRAM, one language-model inference at a time) are concrete engineering anchors for anyone deploying agents next to a compute-hungry application.

Real-world applications:

• Non-player characters and co-playable companions in multiplayer games that speak and act with players.
• Assistive or companion agents on consumer hardware that must run entirely on-device under tight latency and memory budgets.
• Safety design for agents whose ordinary domain language is aggressive or violent, but which must still refuse harm aimed at real people or groups.
• Physical robotics, which the authors note is already drawing on Ally's agentic design principles in Ludi 0.1.

Industry relevance: The paper speaks directly to studios and product teams weighing whether an LLM-driven character can survive live-service constraints, multilingual rollout (English, Korean, and Chinese with locale-specific on-device language and speech models over a two-week beta), and player-facing safety review.

Future Directions

• Closing the gap between offline evaluation and player preference. The team explicitly used disagreements between internal evaluations and player preferences to revise criteria, which raises the question of how well such a framework generalizes to new models, events, and player populations.
• Extending beyond the scoped setting. The reported deployment is a duo mode with one human player on the Sanhok map in 64-player matches, in three languages; broader modes, maps, team sizes, and languages remain open.
• Scaling the correction-based training loop. Teacher-corrected student rollouts were essential because student behavior creates contexts the teacher never visited, so the cost and coverage of generating corrections at larger scale is an open engineering question.
• Generalizing the deployment stack. The authors connect Ally's design to physical robotics through Ludi 0.1, suggesting the combination of bounded tool interface, layered control, context compaction, and safety mechanisms as a template beyond games.

Target Audience

Researchers and engineers working on embodied agents, game AI, LLM agent harnesses, and on-device model deployment will get the most from this paper, as will product and safety teams at game studios considering voice-enabled AI teammates. Readers interested in human-agent interaction and player-centered evaluation methodology will also find the data collection and evaluation-refinement sections relevant. Portions of the paper are truncated in the available content, so some later sections, appendices, and the full training recipe referenced in Figure 5 are not covered here.

Authors’ abstract

We introduce PUBG Ally, an embodied agent for PUBG: BATTLEGROUNDS that can reason, act autonomously, and play alongside players as a voice-enabled teammate. Building such a teammate requires combining two difficult capabilities: it must perceive and respond to a constantly changing game world under strict latency constraints while interacting naturally with players, keeping its speech synchronized with its actions. Ally therefore combines agentic tool use with real-time game control. A language-model agent uses a controlled interface to inspect game information, interpret player speech, maintain context, decide what to say, and issue high-level action choices that steer a faster control layer for movement, combat, and recovery. Because the player's and Ally's speech and actions continually shape each other and the course of the match, training requires data from actual gameplay. We therefore collect data across nearly 39k sessions in which real players play alongside Ally, recording gameplay, player speech, agent decisions, tool use, actions, and player feedback, and use these records for iterative training. To evaluate teammate quality, we use player feedback and preference comparisons to identify gaps between offline evaluations and player preferences, and iteratively refine the evaluation criteria. Deploying Ally in live service further requires low-latency on-device execution and safeguards for player-facing communication, which we address through model compression, context compaction, targeted safety training, runtime guardrails, and memory redaction. During the live service, we surveyed players in 141 countries. Among respondents whose play with Ally was confirmed in game records, positive responses exceeded negative responses by 25.1 percentage points when asked whether they would recommend Ally, with players describing Ally not only as a tool but also as a teammate or companion.