Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments

AI summary

Overview

Research area: Multi-agent systems / generative social simulation, with a companion embodied-AI component using vision-language models in 3D digital twins.

Technical level: Beginner-Friendly. The paper explicitly positions both codebases as "intentionally minimal" educational foundations rather than finished applications, with locally hosted models and no specialized compute requirements.

Scope: The paper presents two small, open, locally runnable simulation codebases — one for LLM-driven multi-agent social simulation on a 2D grid, one for VLM-driven embodied navigation in a 3D USD digital twin — and documents their architecture, prompts, and example runs.

What This Paper Is About

Contemporary LLM- and VLM-driven agent simulation platforms are often large, dependent on remote model services, and hard to run on an ordinary computer, which obscures the basic perception-reasoning-action loop that makes them work. The author releases two deliberately minimal, locally hosted foundations — SD-AgentFoundry-2D and SD-AgentFoundry-3D — as readable starting points for education and rapid prototyping. The goal is accessibility and modifiability, not feature completeness or a validated predictive model.

Key Contributions

1. SD-AgentFoundry-2D: a Python discrete 2D multi-agent environment where agents move on a bounded integer grid, communicate with spatially eligible neighbors, respond to place occupancy and capacity, and encounter spatially localized fire events with positions, intensities, and perception radii. The environment supplies numerical state only, leaving interpretation such as avoidance, warning, coordination, or inaction to the LLM rather than encoding it as simulator rules.
2. SD-AgentFoundry-3D: a 3D single-agent digital-twin environment built on Universal Scene Description (USD) assets, using usd-core for scene access and VTK/PyVista for rendering rather than a heavyweight game engine. A locally hosted VLM receives first-person RGB images and returns natural-language movement instructions.
3. Cross-platform, locally operated tooling: both codebases run on macOS, Windows, and Linux with setup scripts, use Ollama-served local models, centralize configuration in YAML, and log auditable run artifacts (JSONL memory/reasoning records, message logs, PNG frames, pose histories, raw and parsed decisions).
4. Published prompt templates and reproducibility artifacts: Appendix A documents the exact communication prompt, action-decision prompt, and VLM text prompt templates, plus worked examples of assembled prompts.

Main Findings

• Two example runs are documented, not evaluated. The paper states that the representative 2D and 3D runs are "qualitative usage examples, not as controlled evaluations or evidence of predictive validity." No task success rate, no benchmark comparison, and no quantitative behavioral result is reported.
• 2D example configuration: 20 LLM-driven agents; both grid axes range from −25 to 25; a cafe at (−15, 0) with capacity 12 and a library at (15, 0) with capacity 10; fire1 activates at step 3 with intensity 0.8 and radius 15; fire2 at step 5 with intensity 0.5 and radius 8; state recorded after step 7.
• 2D step-7 state: both fires active, 10 agents within at least one perception radius, and 0 agents inside places. Recorded through step 7 were 8 PNG frames, 140 memory/reasoning records, and 88 per-recipient message-delivery records.
• 2D model setup: Ollama serving qwen3:4b-instruct-2507-q4_K_M, temperature 0.2, thinking disabled, maximum 512 response tokens. Each agent makes two LLM calls per step — a communication prompt and an action-decision prompt.
• 3D example configuration: Ollama serving qwen3.6:27b; scene assets/kibou/KIBOU.usd excluding the distant Earth backdrop; task "Find JAXA logo and get close to it"; initial pose (4.0, 20.1) m, yaw 180°, pitch 0°, eye height 1.6 m; camera 512 × 384 pixels with 70° vertical field of view; run length of 10 VLM decisions.
• The 3D loop is not a trained VLA policy. The VLM emits unrestricted natural language, and a deterministic interpreter extracts a translation bearing, body rotation, gaze pitch, or stay command, resolving each movement category from its latest occurrence in the text and converting supported distance units to meters.
• Explicit non-claim about prediction. The paper states that LLM simulations of established human-subject studies can reproduce some findings while also exhibiting systematic distortions, and that such simulations are best treated as instruments for generating hypotheses rather than substitutes for empirical evidence.
• Reproducibility limitation in 2D. Exact repetition of the 2D example requires control of the randomly generated initial positions and personas, for which the present configuration does not expose a seed.

Methodology in Plain English

The author built two small simulators rather than a large platform. In the 2D system, each simulation step runs in a fixed order: every agent identifies which other agents it can talk to (requiring both spatial proximity and a shared area — either both outside all places, or both inside the same place), decides on a message, messages are delivered using the pre-movement neighborhood, each agent decides on an action, and then all movements execute. Memory and received messages are kept in bounded histories and fed into later prompts. Fire events only expose numerical properties and distance to agents inside their perception radius; everyone else must learn about them through messages.

In the 3D system, at each decision point the simulator renders a first-person RGB image, sends it with a dynamically assembled text prompt (task, current pose, bounded recent history) to a VLM, and parses the free-text reply into a bounded action. The interpreter resolves each movement category from its latest mention because reasoning models state conclusions at the end; unreadable responses are logged without inventing a substitute action, and configured limits clamp excessive motion. Inference wall-clock time is kept separate from simulation time so model latency does not alter simulated dynamics. A separate replay path reconstructs overview frames from the log rather than treating the live window as a video recording for auditing purposes.

Why This Matters

Impact on research: The paper argues that large platforms, remote model services, and substantial compute requirements can obscure the fundamental perception-reasoning-communication-action-observation loop. By publishing readable implementations with documented prompts and auditable logs, it lowers the barrier for studying generative social simulation and for comparing LLM- or VLM-specific behavioral signatures — such as movement preferences, hazard sensitivity, communication frequency, warning propagation, clustering, collective place choice, object-recognition errors, and exploration strategies — while holding the scenario fixed.

Real-world applications (as framed by the paper):

• Exploratory studies of communication, evacuation, public policy, urban activity, and collective response to external events.
• Organizational prototyping of services and examination of hypothetical customer or worker interactions.
• Digital-twin experiments on navigation, facility use, human-robot interaction, and crowd behavior.
• Educational use for students learning how generative social simulation is built.

Industry relevance: The project frames agent simulation as an instrument for generating hypotheses and rehearsing possibilities, and states outputs require empirical validation before supporting real-world claims — a useful guardrail for organizations adopting simulation for planning. Its use of USD scenes means organizations with existing 3D assets can substitute their own environments.

Future Directions

1. Controlled comparison studies. The paper suggests holding scenario conditions constant while swapping the underlying model to compare behavioral signatures, which would require controlled random seeds, repeated trials, and validation against appropriate human or observational data.
2. Adding a seed to the 2D configuration. Exact repetition of the current 2D example is not possible without exposing control over randomly generated initial positions and personas.
3. Extending the 3D scene vocabulary. Users can replace the Kibo asset with another USD scene and vary the task instruction, initial pose, camera field of view, movement and gaze limits, recent-history length, model backend, and response interpreter.
4. Connecting to learned low-level control. The paper notes that SD-AgentFoundry-3D cannot by itself learn a new motor skill, generate joint- or end-effector-level control, handle contact-rich manipulation, or close a high-frequency sensorimotor loop; those capabilities would require a VLA or another trained low-level policy, an embodiment-specific interface, and separate physical validation.

Target Audience

Students, educators, researchers, and practitioners who want an accessible, modifiable starting point for LLM-driven social simulation and VLM-driven embodied simulation on ordinary desktop computers. It is best suited to readers interested in learning the mechanics of the perception-reasoning-action loop, building domain-specific extensions, or studying model-dependent agent behavior under a fixed, transparent simulator design — not to readers seeking benchmark results or a validated predictive simulator, since no such evaluation is reported.

Authors’ abstract

Large language models (LLMs) and vision-language models (VLMs) are expanding the range of behaviors that can be represented in agent-based simulations, but many contemporary platforms are difficult to study, modify, or run on ordinary computers. We present two intentionally minimal simulation foundations for education and rapid prototyping. SD-AgentFoundry-2D provides a two-dimensional multi-agent environment in which locally hosted LLM agents move, communicate, respond to place occupancy, and encounter spatially localized fire events. SD-AgentFoundry-3D provides a three-dimensional digital-twin environment in which a locally hosted VLM receives first-person images and produces natural-language movement instructions. Both codebases are designed to run locally on macOS, Windows, and Linux and are deliberately left open to modification rather than developed as finished applications. Together, they offer accessible starting points for learning about generative social simulation and for building domain-specific extensions.