Large Language Model-Based Reward Design for Deep Reinforcement Learning-Driven Autonomous Cyber Defense

AI summary

Overview

Research area: Applied machine learning for cybersecurity — specifically, using a large language model (LLM) to design reward functions that train deep reinforcement learning (DRL) agents for autonomous cyber defense.

Technical level: Intermediate. The paper assumes familiarity with reinforcement learning basics (policies, rewards, actor-critic methods) and general cybersecurity terminology (kill chains, lateral movement, decoys), but it explains its framework clearly enough for readers with a background in one of those areas.

Scope in one sentence: The paper shows that an LLM (Claude Sonnet 4) can translate qualitative descriptions of attacker and defender "personas" into concrete numerical reward tables, and that the resulting DRL-trained defender policies measurably delay attacker impact in a simulated 15-host network.

What This Paper Is About

Training autonomous cyber defense agents with deep reinforcement learning requires a reward function that tells the agent what "good" and "bad" outcomes look like — but hand-crafting those rewards for complex, heterogeneous attack and defense behaviors is cognitively demanding for cyber subject matter experts. This paper asks whether an LLM can take on that reward-design burden by converting plain-language behavioral descriptions (such as "make the attacker more aggressive" or "make the defender more proactive") into usable reward structures. The authors then test whether defense policies trained on those LLM-generated rewards actually perform better against different attacker types in the Cyberwheel simulation environment.

Key Contributions

1. An LLM-assisted reward design workflow. The authors provide Claude Sonnet 4 with contextual information about the Cyberwheel simulation environment and baseline SME-authored reward files, then prompt it to produce modified reward structures for attacker (red) and defender (blue) agents based on qualitative behavioral traits.

2. Explicit reward tables for heterogeneous agent personas. The paper defines and publishes reward structures for an aggressive red agent, a stealthy red agent, and two proactive blue agents (proactive-v1 and proactive-v2), alongside the SME baseline versions — making the LLM's quantitative output directly inspectable.

3. A systematic PPO-based training and evaluation study. Nine experiments cover every combination of three blue personas (baseline, proactive-v1, proactive-v2) and three red personas (baseline, stealthy, aggressive), with policies trained under a documented DRL hyperparameter set and evaluated over 50 episodes of 100 steps each.

4. An empirically motivated mixed-persona defense strategy. Based on time-to-first-impact statistics, the authors conclude that the best overall defender policy is to switch blue personas depending on which red persona is encountered.

Main Findings

• LLM-guided rewards produced effective, differentiated defense behavior. The authors report that "LLM-guided reward designs can lead to effective defense strategies against diverse adversarial behaviors," with proactive blue agents delaying the red agent's impact action against stealthy and aggressive attackers.

• The red agent does not change behavior; only the blue agent's perception changes. Because the red agent is heuristic (not a learning agent), its action frequencies are essentially unchanged across persona pairings. What changes is how the reward structure makes the learning blue agent respond to those actions.

• Decoy placement proportion depends on both blue and red persona. Against the aggressive red agent, proactive-v1 dedicated 39.5% of its actions to decoy placement versus 35.1% against the stealthy red agent. For proactive-v2, the pattern reverses: 34.2% decoy placement against the stealthy red agent versus 22.4% against the aggressive one.

• 95th-percentile time to first impact (worst-case delay), with 50 evaluation episodes of 100 steps:

  • Against the baseline red agent: baseline blue = 15 time steps; proactive-v1 = 11; proactive-v2 = 11.
  • Against the stealthy red agent: proactive-v2 = 18 time steps; proactive-v1 = 13; baseline = 13.
  • Against the aggressive red agent: proactive-v2 = 15 time steps; proactive-v1 = 12; baseline = 10.

• No single blue persona wins everywhere. The baseline blue agent is best against the baseline red agent, while proactive-v2 is best against both the stealthy and aggressive red agents — motivating the mixed strategy recommendation.

• The baseline reward structure is highly decoy-averse. The SME baseline assigns a decoy placement an immediate reward of -20.0, whereas the LLM-generated proactive-v2 assigns -5.0 with a recurring reward of -0.5, and proactive-v1 assigns +20.0 with a recurring +2.0.

Methodology in Plain English

1. Set up a simulation. The work uses Cyberwheel, a fast, scalable, customizable cyber simulation built on networkx graphs containing hosts, subnets, and routers (configured through yaml files). The experiments use the network shown in the paper's Figure 2, described as a 15-host network with one router and three subnets (an earlier description in the paper references an example topology of one router, three subnets, and five hosts).

2. Define the agents. The red (attacker) agent follows a logic-driven kill-chain based on Atomic Red Team and MITRE ATT&CK techniques, with actions including ping sweep, port scan, discovery, lateral movement, privilege escalation, and impact. Its entry host is randomized each episode. The blue (defender) agent uses a cyber deception strategy with three possible actions: deploy a decoy, remove a decoy, or do nothing. Only the blue agent learns; the red agent is heuristic.

3. Characterize behaviors qualitatively. The authors describe an "aggressive" attacker (rushes to privilege escalation and impact, accepting high detection risk) and a "stealthy" attacker (slow escalation, long-duration foothold, avoids detection). For the defender, they describe "proactive-v1" (frequent actions, high incentive for decoy placement) and "proactive-v2" (proactive but under resource constraints).

4. Ask the LLM to convert words into numbers. Claude Sonnet 4 is first given context about the Cyberwheel environment, then prompted with behavioral requests — for example, "How can the attacker (red) agent be made more aggressive?" — alongside the baseline yaml reward file, and asked to return modified rewards. This produces the reward tables the paper reports.

5. Train and evaluate with PPO. The blue agent is trained with proximal policy optimization under a clipped surrogate objective (actor-critic), using the hyperparameters in the paper's Table 1 (learning rates 2.5 × 10⁻⁴, discount factor γ = 0.99, GAE λ = 0.95, 4 minibatches, 4 update epochs, clipping coefficient ε = 0.2, entropy coefficient 0.01, value function coefficient 0.5, maximum gradient norm 0.5, learning-rate annealing enabled). Rewards at each step combine the blue agent's reward, the red agent's reward, and recurring costs accumulated over the episode.

6. Measure success as delay. Nine persona combinations are trained. The key performance metric is the complementary empirical cumulative distribution of time to first impact — the number of steps before the red agent first succeeds at the impact action — with the 95th percentile treated as a robust worst-case lower bound.

Why This Matters

Impact on research. The paper demonstrates that LLM4RL-style reward design (using an LLM to supplement RL training on a non-language task) can be applied to a safety-critical, adversarial domain where reward engineering is a recognized bottleneck. It also provides a concrete, inspectable artifact — published reward tables — which makes the LLM's contribution auditable rather than opaque, and it shows that the LLM can encode behavioral nuance (persistent vs. bursty incentives) purely through reward numbers.

Real-world applications:

• Enterprise network defense: automatically generating decoy-deployment policies tailored to different observed attacker profiles, reducing analyst workload.
• Security operations centers (SOCs): using persona-matched defensive posture switching as a decision aid when threat intelligence indicates a shift from opportunistic to stealthy or aggressive intrusion activity.
• Critical infrastructure protection: supporting secure-by-design and resilient AI-enabled tools, which the paper links to the priorities stated in America's AI Action Plan.
• Training and simulation: rapidly standing up attacker/defender scenario variations in cyber ranges without requiring a subject matter expert to hand-tune every reward file.

Industry relevance. Reward design is one of the most labor-intensive parts of deploying reinforcement learning in production security tooling. A workflow where an expert writes a sentence instead of a table — and where the resulting table is reviewable and refinable — lowers the barrier to experimenting with autonomous defense, which matters for organizations with limited security staff.

Future Directions

1. Make the red agent a learning agent. Currently only the blue agent learns; letting the attacker co-adapt would test whether LLM-designed rewards remain effective against genuinely responsive adversaries.

2. Scale to multiple agents and larger networks. The paper lists including multiple red and blue agents and scaling up the state space of the cyber host network as next steps beyond the 15-host configuration used here.

3. Move to an LLM-in-the-loop design. Rather than a one-shot reward prompt, the LLM would suggest a reward structure, the agent would train, and the LLM would iteratively refine the rewards based on measured performance.

4. Address robustness and explainability. The authors argue that keeping an LLM in the process — either for initial design or in the loop — could make DRL-based autonomous cyber defense agents more robust to changing adversary tactics and make their actions more explainable.

Target Audience

Researchers and practitioners at the intersection of reinforcement learning and cybersecurity — particularly those working on autonomous cyber defense, cyber deception, or adversarial simulation. It is also relevant to security engineers evaluating whether LLM-assisted tooling can reduce the expert effort required to specify objectives for learned security agents, and to readers interested in concrete examples of LLM4RL reward design outside of robotics and control benchmarks.

Authors’ abstract

Designing rewards for autonomous cyber attack and defense learning agents in a complex, dynamic environment is a challenging task for subject matter experts. We propose a large language model (LLM)-based reward design approach to generate autonomous cyber defense policies in a deep reinforcement learning (DRL)-driven experimental simulation environment. Multiple attack and defense agent personas were crafted, reflecting heterogeneity in agent actions, to generate LLM-guided reward designs where the LLM was first provided with contextual cyber simulation environment information. These reward structures were then utilized within a DRL-driven attack-defense simulation environment to learn an ensemble of cyber defense policies. Our results suggest that LLM-guided reward designs can lead to effective defense strategies against diverse adversarial behaviors.