Think, Speak, Decide: Language-Augmented Multi-Agent Reinforcement Learning for Economic Decision-Making

AI summary

Overview

Research area: Multi-agent reinforcement learning (MARL) combined with large language models (LLMs) for economic decision-making, specifically tax policy and household behavior simulated in the TaxAI environment.

Technical level: Advanced. The paper assumes familiarity with partially observable Markov games, centralized training with decentralized execution (CTDE), MADDPG, LLM prompting, and embedding-based retrieval.

Scope: The paper proposes LAMP (Language-Augmented Multi-Agent Policy), a Think–Speak–Decide pipeline that injects LLM-generated reasoning, peer messaging, and reflection into a MARL policy, and evaluates it against conventional and LLM-only baselines in three economic scenarios.

What This Paper Is About

Economic decisions depend on structured numbers such as prices, wages, and taxes, but also on unstructured language such as peer dialogue, public statements, and media narratives. Standard MARL methods assume clean, structured communication and ignore language, while LLM-only approaches generate actions or simulate behavior without systematically optimizing an agent's policy. The paper's goal is to integrate LLM reasoning and dialogue into a MARL training loop so that economic agents can interpret and act on natural-language signals while still being optimized by reinforcement learning.

Key Contributions

1. A framework, LAMP, that models the role of natural language in economic decision-making by augmenting multi-agent policies with LLM-derived reasoning and reflection, aimed at bringing simulations closer to real-world settings.
2. A Think–Speak–Decide mechanism that explicitly structures how agents extract short-term shocks and long-term trends, craft and exchange strategic messages, update beliefs and trust about peers, and fuse all of this into policy optimization.
3. Empirical evidence that LAMP surpasses MARL and LLM-only baselines in language-guided decision performance across three economic scenarios, in the paper's words with gains in cumulative return (+63.5%, +34.0%) and robustness (+18.8%, +59.4%), plus interpretability.
4. Ablation and interpretability analysis isolating the contribution of each component (Speak, experience pool, long-term reasoning, short-term reasoning, and trigger timing) and showing that the framework's stored reasoning traces form an auditable knowledge base explaining why strategies were chosen.

Main Findings

• Headline comparison against language baselines: In S1 (Economic Stability), using the same LLM backbone (Qwen2.5-72B-Instruct-INT4) and prompt budget, LAMP surpasses the strongest language baseline (ReAct) with +14.8% higher welfare and +14.5% higher reward, while reducing consumption and labor. Under distribution shift, welfare gains in S2 and S3 are +1.0% and +10.4%, and reward gains are +18.1% and +16.0%, again with reductions in consumption and labor.

• Headline comparison against non-language baselines: In S1, versus the strongest non-language baseline (Rule-Based), welfare improves +12.3% and reward +12.1%; versus numeric MARL (MADDPG), gains reach +118.8% and +63.5%. Versus Rule-Based, LAMP uses –27.9% consumption and –44.9% labor; versus MADDPG, –56.8% and –60.0%. The paper attributes higher welfare to efficiency rather than brute-force spending or overwork.

• Absolute S1 results (Table 2): LAMP records average reward 8.52 ± 0.13, social welfare 2.56e+03 ± 3.77e+01, consumption 2.30e+05 ± 7.52e+04, and labor 3.13e+05 ± 8.46e+04. For comparison, Rule-Based reaches 7.60 ± 0.33 reward and 2.28e+03 ± 9.99e+01 welfare; MADDPG 5.21 ± 0.16 and 1.17e+03 ± 5.51e+02; Random 6.53 ± 0.35 and 1.96e+03 ± 1.06e+02; LLM-Only 6.35 ± 0.32 and 1.90e+03 ± 9.56e+01; CoT 6.75 ± 0.34 and 2.03e+03 ± 1.03e+02; ReAct 7.44 ± 0.26 and 2.23e+03 ± 7.92e+01; Reflexion 6.59 ± 0.31 and 1.98e+03 ± 9.16e+01.

• Speak module matters for coordination: Removing Speak causes a 1.2% welfare drop alongside sharp increases in labor and consumption. The paper interprets this as agents compensating through brute-force effort when strategic communication is unavailable, and notes that the LLM revises beliefs toward demand fragility and recommends restraint such as moderating labor expansion when inequality widens and wages are low.

• Experience pool drives stability and efficiency: Removing it cuts social welfare by 50.9% and average household reward by 0.8%, while labor rises 43.6% and consumption surges 122.4%; stable simulated years fall by 50.2%. Table 3 lists the same setting at 1.25e+03 social welfare, 5.12e+05 consumption, 4.50e+05 labor, and 1.50e+02 years.

• Long-term reasoning prevents myopia: Removing it drops average household reward by 37.7% and reduces stable years from 300 to 219. Table 3 shows average reward falling to 5.31 and social welfare to 1.15e+03.

• Short-term reasoning supports shock response: Disabling it has a moderate effect on final returns (–3.99% reward) but harms efficiency: labor rises 67.7%, consumption 52.7%, and stable years fall from 300 to 208.

• Trigger timing is critical: Random triggers keep welfare similar but collapse efficiency: labor increases 81.9%, consumption 51.2%, and stable years drop from 300 to 141. Table 3 shows average reward unchanged at 8.52 (–0%) but social welfare at 1.19e+03 (–53%).

• Adaptive policy shifts appear in LLM outputs: On detecting rising inequality (top 10% volatility widening, bottom 50% declining), the LLM revised its earlier "work more" stance toward slightly reducing work hours, increasing savings, delaying non-essential spending, investing in skills, and publicly supporting progressive taxation and minimum wages.

Methodology in Plain English

The authors build on TaxAI, a dynamic economic simulator previously calibrated with real-world data, which models heterogeneous households interacting with a government. They add language to each household's observation by passing inputs through an LLM to produce a textual message, then through an embedding model that maps the text into a numeric vector.

The decision problem is framed as a partially observable Markov game with a government agent plus household agents. The government sets parameters of marginal income-tax and asset-tax schedules plus an expenditure-to-output ratio, while each household chooses a savings rate and labor supply.

LAMP then runs in three stages every episode:

• Think. A shared LLM-generated news service converts global numerical signals into two kinds of text. Long-term news is produced at fixed checkpoints and captures structural trends. Short-term shock news is produced whenever a key indicator — the wealth Gini, social welfare, or per-capita GDP — moves by more than a threshold. Each agent then infers its economic status (good, neutral, or poor) and writes a private reasoning trace. High-reward reasoning trajectories are ranked and cached: each agent stores its top trajectories in a short-term buffer, and at each long-term checkpoint the system collects the top trajectories across all agents into a long-term FAISS index. Before the next long-term reasoning step, an agent retrieves the nearest neighbors from that index using its current observation and merges them with its short-term buffer to prompt the LLM.

• Speak. Guided by the reasoning, the LLM generates three candidate public statements per agent, and a lightweight self-attention selector scores them into a distribution from which one statement is sampled and broadcast. Each agent then uses a reflection module to interpret the broadcast messages, producing an estimated wealth tier for each peer (low, mid, or high), a numeric belief confidence in the range 0 to 10, and a brief self-reflection. These feed back into the selector and the policy, closing a loop between reasoning, dialogue, and coordination.

• Decide. All text (private reasoning and reflection) is encoded by a frozen text encoder, pooled into a fixed-length vector, and passed through a small trainable projection for dimensionality reduction and alignment. This language embedding is concatenated with numeric observations to form the state. The system uses a standard MADDPG setup: a centralized critic minimizes Bellman error, and decentralized actors update policies by maximizing expected Q-values via deterministic policy gradients. Gradients do not flow into the text encoder unless noted.

Evaluation uses five metrics: average household reward, social welfare, total consumption, total labor, and years (simulated years before collapse, capped at 300). Baselines span a Random policy, a Rule-Based economic method, MADDPG, and LLM-based methods: Only-LLM, CoT, ReAct, and Reflexion. Three scenarios are used: S1 Economic Stability (matches training conditions), S2 Economic Slowdown (moderate shift, reduced growth and mild market stress), and S3 Crisis Shock (large coupled shift for robustness).

Why This Matters

Impact on research. The paper targets a gap between MARL, which optimizes policies but assumes structured communication, and LLM-based economic work, which processes language but typically generates actions or simulates behavior without policy optimization. LAMP shows one way to combine both: LLM outputs become structured features inside an RL training loop, and the resulting reasoning traces serve as an audit trail. It also reports that purely data-driven MARL must search from scratch in environments with many interdependent, frequently changing variables, while LLM reasoning supplies compact high-value signals.

Real-world applications:

• Tax policy design and evaluation, where governments must weigh household welfare, inequality, and growth under noisy public debate.
• Household financial planning and labor-supply advice under income shocks and shifting media narratives.
• Analysis of how peer dialogue and public statements shape economic expectations and coordinated behavior.
• Stress-testing policy rules against slowdowns and crisis shocks, using the S2 and S3 setups as robustness probes.

Industry relevance. Any setting where decisions depend on both numbers and text is a candidate: financial trading and market pricing, firm pricing strategy in competitive markets, policy advisory and regulatory analysis, and simulation tools used by governments or research institutions. The code is released at https://github.com/hey0223/LAMP.

Future Directions

• Scaling and generalization beyond TaxAI: The paper evaluates only in TaxAI across three scenarios; whether the Think–Speak–Decide pipeline transfers to other economic simulators or other MARL environments is not established.
• Testing other LLM backbones and MARL algorithms at scale: The main results use Qwen2.5-72B-Instruct-INT4, with comparisons of other language models and MARL algorithms deferred to the appendix. How sensitive the approach is to backbone choice, prompt design, and cost remains an open question raised by the design.
• Handling deceptive or adversarial language: The paper motivates LAMP partly by noting that real language can be noisy and sometimes deceptive, but the reported experiments do not include adversarial communicators; testing whether reflection and trust updates hold up under manipulation is a natural next step.
• Treating the frozen text encoder and learned projection as a design choice to revisit: Because gradients do not flow into the encoder for stability and only the projection is updated, whether joint fine-tuning improves language-augmented policies is left open.

Target Audience

Researchers and practitioners working at the intersection of multi-agent reinforcement learning and large language models, especially those applying RL to economics, public policy, taxation, or market simulation. It will also interest economists who want to see how language-based signals such as public opinion and peer dialogue can be modeled computationally, and engineers building LLM-plus-RL systems that need interpretable decision traces rather than black-box actions.

Authors’ abstract

Economic decision-making depends not only on structured signals such as prices and taxes, but also on unstructured language, including peer dialogue and media narratives. While multi-agent reinforcement learning (MARL) has shown promise in optimizing economic decisions, it struggles with the semantic ambiguity and contextual richness of language. We propose LAMP (Language-Augmented Multi-Agent Policy), a framework that integrates language into economic decision-making and narrows the gap to real-world settings. LAMP follows a Think-Speak-Decide pipeline: (1) Think interprets numerical observations to extract short-term shocks and long-term trends, caching high-value reasoning trajectories; (2) Speak crafts and exchanges strategic messages based on reasoning, updating beliefs by parsing peer communications; and (3) Decide fuses numerical data, reasoning, and reflections into a MARL policy to optimize language-augmented decision-making. Experiments in economic simulation show that LAMP outperforms both MARL and LLM-only baselines in cumulative return (+63.5%, +34.0%), robustness (+18.8%, +59.4%), and interpretability. These results demonstrate the potential of language-augmented policies to deliver more effective and robust economic strategies.