Evolution in Simulation: AI-Agent School with Dual Memory for High-Fidelity Educational Dynamics

AI summary

Overview

• Research area: Multi-agent LLM simulation of educational settings; agent-based modeling of school dynamics; "Computational Education Science."
• Technical level: Intermediate. The core ideas are readable without deep mathematics, but familiarity with LLM agents, retrieval-augmented generation, and ROUGE-style evaluation helps.
• Scope: The paper proposes the AI-Agent School (AAS), a multi-agent simulated school built on a self-evolving "Zero-Exp" mechanism with a dual experience/knowledge memory base, and evaluates it on a 5-day, 50-agent expert-curated dataset. (The paper uses the expansion "AI-Agent School" in the abstract and "Agent-based Learning Simulation" once in related work.)

What This Paper Is About

Existing LLM-based educational agents are limited by a lack of systematic modeling of the teaching process and by difficulty in simulating the diverse behaviors and interactions of real educational participants. The paper's goal is to build a simulated school in which teacher and student agents start from a near zero-experience state and autonomously evolve, through a repeated "experience-reflection-optimization" cycle, into agents whose behavior and interactions resemble those found in physical schools. The authors position this as a step from the "Era of Experience" to the "Era of Simulation," producing high-fidelity behavioral and interaction data for education.

Key Contributions

1. The AAS multi-agent educational simulation system. A simulated school environment (built with Tiled and Cocos) containing 25 areas—classrooms, libraries, laboratories, sports fields, and others—that supports teacher agents and student agents and is designed to capture teacher-student relationships, peer interaction, and environmental influence across multiple roles and over time.
2. The Zero-Exp mechanism. A mechanism intended to address data scarcity and role-behavior consistency by guiding agents from a zero-experience state toward expert-level behavior using a small set of initialization parameters. It centers on a dual memory base that separates an Experience Base (lived events and interactions) from a Knowledge Base (structured role knowledge, facts, and principles), each further divided into Short-term and Long-term components, with retrieval ranked by cosine similarity and short-term memory prioritized.
3. A structured, expert-curated evaluation setup. A Standard Group dataset (ID 0) generated with QwQ-32B role settings and Gemini-2.5-Pro action sequences, then modified and reordered by educational experts, serving as ground truth. The study compares nine memory configurations across three agent LLMs (GPT-4o, Qwen3-235B-A22B, Qwen3-8B), using automated ROUGE-L scoring and blind human voting by nine educational experts.
4. An explicit framing of a new research paradigm. The authors present the work as laying a theoretical and technical foundation for "Computational Education Science," next-generation educational systems, teacher training platforms, and educational policy simulation tools.

Main Findings

• The full memory model achieves the highest automated scores. Configuration ID 1 (dual Experience/Knowledge base plus the Short-term/Long-term hierarchy) reaches the highest ROUGE-L scores, with peaks described as around 0.51–0.55 in the later stages of the simulation. Concrete endpoints: with GPT-4o, ID 1 rises from 0.11 at the 5% checkpoint to 0.55 at the 95% checkpoint and 0.53 at 100%; with Qwen3-235B-A22B, ID 1 reaches 0.58 at 95% and 0.56 at 100%; with Qwen3-8B, ID 1 reaches 0.50 at 95% and 0.48 at 100%.
• Agents show a learning curve, then plateau. Most configurations show an initial increase in ROUGE-L as agents accumulate experience and knowledge, with performance tending to stabilize or fluctuate in later stages rather than rising indefinitely.
• External memory provides a large benefit. Comparing ID 9 (Context Only, no external memory base) with any configuration using an external memory base (IDs 1–8) shows a substantial performance gap. ID 9 stays near 0.24–0.28 in the later GPT-4o checkpoints, versus 0.53 for ID 1 at 100%.
• Separating experience from knowledge helps. The dual structure outperforms a unified one: ID 1 consistently outperforms ID 2, and ID 3 generally performs better than ID 4, though the ID 3–ID 4 gap is smaller in some phases.
• The Short-term/Long-term hierarchy helps. ID 1 outperforms ID 3 and ID 2 outperforms ID 4; comparisons of single-base configurations (ID 5 vs. ID 7 and ID 6 vs. ID 8) consistently favor the ST/LT hierarchy. The authors attribute this to prioritizing recently salient memories.
• Experts increasingly prefer the fully evolved agents. In the blind human evaluation (agents acted by GPT-4o), the expert-curated Standard Group (ID 0) starts dominant at 92% of votes at the 10% checkpoint but declines to the 44–48% range from 60% onward. The full model (ID 1) climbs from 2% at 10% to 42% at 60%, 46% at 70%, and 44% at 80–100%, approaching and stabilizing near the Standard Group. The RAG Only (ID 4) and Context Only (ID 9) groups stay low throughout (ID 4 ranges from 2% to 12% across checkpoints; ID 9 from 4% to 10%).
• The simulation generated large volumes of behavioral data. Recorded action counts include 12,873 teacher Teaching Practice actions, 795 Teaching Reflection, 412 Other Guidance, 9,499 student Peer Learning/Interaction actions, 8,532 Extracurricular Activities, 2,476 Laboratory Work, 1,708 Self-Directed Learning, and 681 Classroom Learning.

Methodology in Plain English

The authors built a simulated school rather than running a study in a real one. The map and interaction framework were created with Tiled and Cocos, covering 25 areas. Role profiles for teacher and student agents were generated with the QwQ-32B model using structured prompts (Appendix A), specifying details such as personality traits, teaching philosophy, academic performance, and learning style.

Each agent is given a layered memory: a working memory corresponding to the context window, plus a dual external memory base stored in a vector database. That base is split into an Experience Base (what happened to the agent) and a Knowledge Base (facts, role knowledge, and learned principles), and each of those is split into Long-term (everything accumulated) and Short-term (the subset the agent judges most salient right now). Retrieval uses cosine similarity between the current query and stored memory vectors.

At each simulation step, the agent receives a system prompt with its role setting and a user prompt with the current location and interaction context, plus retrieved memories, with short-term memory consulted first. After acting, the agent updates its experience and knowledge memories, updates its role setting based on reflection, and sometimes updates the environment state (for example, moving from a classroom to a teacher's office). This loop is the Zero-Exp mechanism.

To create ground truth, the authors generated initial role settings for 10 teachers and 40 students, had experts revise them while the map and schedule were developed in parallel, used Gemini-2.5-Pro to generate an initial 5-day sequence of movements, dialogues, and activities (classroom teaching data came from real classrooms), and then had educational experts modify, reorganize, and reorder the output. The result is ID 0, the Standard Group.

Evaluation used nine configurations defined by four switches—Experience Base on/off, Knowledge Base on/off, memory base structure (Dual, Unified, or None), and ST/LT hierarchy on/off. Three LLMs acted as agents: GPT-4o, Qwen3-235B-A22B, and Qwen3-8B. Automated scoring used average ROUGE-L against reference answers at every 5% interval of the simulation data. Human evaluation covered four groups (ID 0, ID 1, ID 4, ID 9) at every 10% increment, extracting one QA pair per agent (10 teachers and 40 students) for 50 QA pairs per checkpoint and 500 QA pairs per group; nine educational experts voted blindly in randomized order on which answer best reflected a realistic response.

Why This Matters

Impact on research. The paper argues that current educational agents suffer from fragmented teaching-process modeling and poor simulation of diverse participants, and it offers a testable architecture—dual memory plus hierarchical short-term/long-term retrieval—for multi-agent evolution. The convergence of the full model's expert preference with the expert-curated ground truth over time provides a qualitative benchmark for judging whether simulated agents are becoming more realistic, and the framework is presented as a foundation for educational digital twins and for "Computational Education Science."

Real-world applications (as framed by the authors):

• Teacher training platforms, where teacher agents and simulated classrooms let educators rehearse and iterate on practice.
• Educational policy simulation tools that model how changes propagate through a school.
• Generation of valuable educational interaction data for analysis, where real classroom data is scarce.
• Personalized learning pathway design and automated feedback systems, listed as next steps for applying the generated high-fidelity data.

Industry relevance. The work is relevant to developers of LLM agent frameworks, educational technology companies building tutoring or classroom simulation products, and organizations that need to accumulate expertise quickly through simulated experience—the authors note the mechanism could have impact on "other industries that require rapid accumulation of experience." It also speaks to the broader enterprise problem of giving agents persistent, role-consistent memory beyond the context window.

Future Directions

• Scaling. The current simulation covers only 50 agents (10 teachers and 40 students) over a 5-day period; moving to a full school with hundreds or thousands of agents over longer durations presents acknowledged computational and design challenges.
• Adding multimodal perception. Because the agents rely on LLMs without Vision-Language Models, they cannot visually perceive the environment or non-verbal cues from other agents; incorporating VLMs or other multimodal models is proposed to improve situated interaction.
• Improving the evolution machinery. The authors propose developing more sophisticated reflection and optimization mechanisms within Zero-Exp to accelerate and refine agent evolution, and exploring alternative or hybrid LLM architectures for stronger reasoning.
• Applying and generalizing the output. Open questions include using the generated data for personalized learning pathway design or automated feedback, and extending the Zero-Exp mechanism and AAS framework to other domains that require accumulating and using complex experience. A further acknowledged open issue is dependence on the quality and diversity of the initial expert-curated dataset and on the capabilities and biases of the underlying LLMs.

Target Audience

This paper is most useful to researchers working on LLM-based multi-agent simulation and agent memory architectures; educational researchers and learning scientists interested in computational modeling of teaching and learning; developers of educational technology, intelligent tutoring, and teacher-training tools; and practitioners exploring digital twins of human-centric systems. Readers seeking a fully quantitative benchmark against real classroom outcomes will not find one here—the evaluation is ROUGE-L against an expert-curated reference set plus blind expert preference voting, and the authors state that many nuances of real educational settings may not be perfectly replicated.

Authors’ abstract

Large language models (LLMs) based Agents are increasingly pivotal in simulating and understanding complex human systems and interactions. We propose the AI-Agent School (AAS) system, built around a self-evolving mechanism that leverages agents for simulating complex educational dynamics. Addressing the fragmented issues in teaching process modeling and the limitations of agents performance in simulating diverse educational participants, AAS constructs the Zero-Exp strategy, employs a continuous "experience-reflection-optimization" cycle, grounded in a dual memory base comprising experience and knowledge bases and incorporating short-term and long-term memory components. Through this mechanism, agents autonomously evolve via situated interactions within diverse simulated school scenarios. This evolution enables agents to more accurately model the nuanced, multi-faceted teacher-student engagements and underlying learning processes found in physical schools. Experiment confirms that AAS can effectively simulate intricate educational dynamics and is effective in fostering advanced agent cognitive abilities, providing a foundational stepping stone from the "Era of Experience" to the "Era of Simulation" by generating high-fidelity behavioral and interaction data.