A-LAMP: Agentic LLM-Based Framework for Automated MDP Modeling and Policy Generation

AI summary

Overview

• Research area: Reinforcement learning automation, LLM-based multi-agent systems, and Markov decision process (MDP) modeling.
• Technical level: Intermediate (assumes familiarity with MDP concepts, RL training loops such as DQN, and LLM agent pipelines).
• Scope: The paper introduces A-LAMP, a modular multi-agent LLM framework that converts free-form natural language task descriptions into a formal MDP formulation, executable RL environment code, and a trained policy.

What This Paper Is About

Applying reinforcement learning to real tasks normally requires a human expert to translate an informal description into a formal MDP, implement an executable environment, and then train a policy — a process the paper describes as time-consuming, error-prone, and expertise-intensive. The authors' goal is to automate that entire chain end-to-end with LLM agents while keeping each step inspectable and verifiable, so that semantic alignment is preserved from task description to trained policy. They argue that single large models like GPT-4o fail at this because modeling errors, fragile code, and misaligned objectives break the pipeline.

Key Contributions

1. Enhanced MDP modeling and policy generation capability: A-LAMP decomposes MDP formulation and policy generation into specialized LLM agents, producing what the authors report as more reliable and consistent policies than a single large LLM such as GPT-4o.
2. Transparency and interpretability: Each MDP component — inferred objectives, decision variables, constraints — is modularized into either a human-readable description or an equation-level formulation, letting experts inspect, validate, and refine the modeling process at any stage.
3. Adaptability across tasks and environments: Because the framework regenerates MDPs and policies from natural language descriptions, new tasks or modified existing tasks can be handled by automated regeneration rather than full manual re-engineering.
4. Productivity gains via automation: A-LAMP automates labor-intensive steps including environment construction, reward specification, and policy initialization, lowering the expertise barrier for RL deployment.

Main Findings

• Highest policy generation success without error correction: Using five benchmark tasks (Cart-pole, Mountain-car, Wireless, Drone-delivery, Inventory-management) with 20 trials each, A-LAMP without the error correction module achieved the highest policy generation success rate across all tasks, with the advantage most pronounced in tasks requiring custom environment generation.
• Roughly double the success of single-model methods on custom-environment tasks: On the wireless, drone-delivery, and inventory-management tasks, A-LAMP is reported to attain nearly twice the policy generation success rate of single-model methods. For example, on drone-delivery, A-LAMP with error correction reached 0.45 policy generation success versus 0.10 for GPT-4o and 0.00 for Gemma3-27B alone; on inventory-management it reached 0.30 versus 0.05 for GPT-4o and 0.00 for Gemma3-27B alone.
• A lightweight variant built on a smaller model approaches a much larger model: Light A-LAMP built on Gemma3-27B substantially outperformed Gemma3-27B alone and approached GPT-4o's performance, which the authors use to argue the gain comes from the framework rather than model scale.
• Error correction gives additional benefit on hard tasks: With the error correction module enabled on the three custom-environment tasks (wireless, drone-delivery, inventory-management), performance improved further compared to A-LAMP without error correction.
• Failure analysis identifies three mechanisms: Across 460 total trials, the paper reports that A-LAMP (1) removes spurious "code succeeds but MDP is wrong" outcomes — entirely eliminated in wireless and reduced by 37.5% in drone-delivery; (2) reduces the complete failure mode where neither modeling nor coding succeeds — by 60% in drone-delivery and eliminated entirely (100%) in inventory-management; and (3) stabilizes training, converting 91% of previously training-stage failures in cart-pole and 58% in mountain-car into successful policies.
• Generated policies preserve task optimality: In the wireless case study, the wireless task was designed so that greedy scheduling is optimal; the DQN policy trained on A-LAMP-generated code closely approached the performance and stability of the greedy scheduler, indicating both executability and correctness.
• Modeling success alone is not the bottleneck: The normalized failure distributions in the appendix show that for successful policy generation cases, MDP modeling succeeded in 1.00 of cases across every task, while failures spread across combinations of modeling, coding, and training stages.

Methodology in Plain English

A-LAMP mirrors how a human expert would approach a new task, splitting the work into three phases and assigning each step to a dedicated LLM agent.

In the abstract idea phase, a parameter agent first extracts the key parameters that define the task scope (for instance, the number of users), then an objective agent extracts the goal statement (for instance, maximizing system throughput). With those in hand, a variable agent identifies decision variables (things the agent can control) and system variables (things that describe the environment), and a constraint agent encodes the feasibility rules. The paper orders these steps by dependency: parameters → objectives → decision variables → constraints → modeling → coding.

In the formulation phase, a modeling agent turns the extracted objective into a standard MDP objective of maximizing expected cumulative reward over a policy, and converts constraints into explicit mathematical forms. A state-action-reward (SAR) agent then defines states, actions, and rewards concretely while preserving internal consistency.

Because the first two phases involve reasoning from long inputs into short extracts, where errors are hard to spot, the framework attaches an error correction module adopted from prior work by AhmadiTeshnizi et al. (2024). Each agent checks its own extraction, assigns itself a self-confidence score, and re-examines its output if the score falls below a threshold; if ambiguity persists, it can issue a clarification request to a human expert.

In the coding phase, an environment agent defines dynamics such as the channel model, path loss and fading, and termination conditions such as a time horizon, while a coding agent writes runnable Gym-style code with environment classes, reset and step functions, and RL training loops. A code executor trains the policy, and a feedback loop between the coding agent and the executor automatically debugs run-time errors.

For evaluation, every task is specified only by a free-form natural language description. The authors measure three things: modeling success rate (whether extracted state, action, and reward are logically correct and complete), coding success rate (whether generated code runs without syntax errors in standard Python), and policy generation success rate (whether RL training converges to a reward-maximizing policy meeting the task objectives). DQN is used as the RL algorithm for all methods and tasks to keep the algorithm backbone constant. Four configurations are compared: A-LAMP with GPT-4o, Light A-LAMP with Gemma3-27B, and single-model baselines of GPT-4o and Gemma3-27B. For the single-model baselines, MDP modeling success is assessed manually by checking consistency of state, action space, and reward definitions inside the generated code, since no intermediate modeling output exists.

Why This Matters

Impact on research. The paper argues that structured multi-agent decomposition — not raw model scale — is the key to reliable RL automation. The failure analysis provides stage-level evidence for where decomposition helps: better modeling reliability, fewer meaningless coding-only outcomes, and more stable training dynamics. It also positions agentic LLM pipelines as a bridge between informal task descriptions and deployable RL policies in a way that remains inspectable by humans.

Real-world applications (as named or exemplified in the paper):

• Network resource scheduling, illustrated by the multi-user wireless scheduling task with Shannon capacity, path loss, and shadowing.
• Industrial automation, cited as a domain where manual MDP construction is a barrier to RL adoption.
• Supply chain optimization, illustrated by the retail inventory-management task with Poisson demand and per-item cost structures.
• Robotics and autonomous control, cited among domains where deep RL has shown strong performance and where the paper's drone-delivery grid-world task provides a planning-under-energy-constraints example.

Industry relevance. The paper claims A-LAMP improves productivity and lowers the expertise barrier for RL deployment by automating environment construction, reward specification, and policy initialization. It also argues the framework reduces re-engineering cost when task objectives change — the paper's example being a switch from maximizing performance to minimizing energy consumption in the same wireless network.

Future Directions

• Strengthening the coding stage. The authors state that although decomposition ensures correctness in MDP modeling, coding remains fragile, and syntactic or structural mismatches can still hinder executability.
• Finer-grained coding agents. They propose dedicated modules for environment construction and training loop generation as a way to improve robustness and reduce debugging costs.
• Adaptive hyperparameter tuning and validation. Adding adaptive mechanisms for these would further generalize the framework.
• Domain-informed priors and structured knowledge integration. The authors identify this as another route toward generalizing A-LAMP to complex domains.

Target Audience

Researchers and practitioners working at the intersection of reinforcement learning and LLM agents, especially those interested in automating MDP formulation and environment construction. It is also relevant to applied RL engineers in networking, industrial automation, supply chain, and robotics who face repeated manual reformulation when task objectives or parameters change, and to readers tracking multi-agent LLM orchestration for structured, verifiable workflows. Readers should be comfortable with MDP terminology and RL training concepts to follow the evaluation and failure analysis.

Note on reporting: the paper does not report wall-clock runtimes, token costs, or compute budgets for any method. A-LAMP with the error correction module was evaluated only on wireless, drone-delivery, and inventory-management; results for cart-pole and mountain-car under that configuration are marked as not reported in Table 1.

Authors’ abstract

Applying reinforcement learning (RL) to real-world tasks requires converting informal descriptions into a formal Markov decision process (MDP), implementing an executable environment, and training a policy agent. Automating this process is challenging due to modeling errors, fragile code, and misaligned objectives, which often impede policy training. We introduce an agentic large language model (LLM)-based framework for automated MDP modeling and policy generation (A-LAMP), that automatically translates free-form natural language task descriptions into an MDP formulation and trained policy. The framework decomposes modeling, coding, and training into verifiable stages, ensuring semantic alignment throughout the pipeline. Across both classic control and custom RL domains, A-LAMP consistently achieves higher policy generation capability than a single state-of-the-art LLM model. Notably, even its lightweight variant, which is built on smaller language models, approaches the performance of much larger models. Failure analysis reveals why these improvements occur. In addition, a case study also demonstrates that A-LAMP generates environments and policies that preserve the task's optimality, confirming its correctness and reliability.