Position: It is Time to Virtualize Foundation Models with a Self-evolving Operating System Layer

AI summary

Overview

Research area: AI systems architecture — specifically the systems/infrastructure layer for compound (multi-agent, tool-using) foundation model applications.

Technical level: Intermediate. The paper is conceptual rather than mathematical, but it assumes familiarity with operating-system concepts (virtual memory, hypervisors, privileged vs. sensitive instructions) and with current LLM agent stacks (LangChain, AutoGen, MCP, context windows, retrieval-augmented generation).

Scope: A position paper arguing that the AI field needs a "Foundation Model Operating System" (FMOS) that virtualizes foundation model interactions behind a stable abstraction, analogous to how operating systems and virtual machine monitors abstracted physical hardware.

What This Paper Is About

AI applications have moved from calling a single foundation model to running compound systems of agents, tools, and multiple models — but every framework (LangChain, AutoGen, Claude Code, Cursor) bakes in its own private runtime for state, memory, budgets, and safety checks, making behavior non-portable and governance brittle. The authors argue this mirrors computing before operating systems, when every program reimplemented its own basic services. Their goal is to define the missing system layer: a Foundation Model Operating System (FMOS) exposing Virtual Foundation Models (VFMs) — the illusion of a dedicated, trustworthy, effectively unbounded model instance — while the FMOS handles knowledge, model routing, verification, and resource allocation underneath.

Key Contributions

1. A formal position that compound agentic systems require a distinct system layer. The paper contends that MCP and A2A solve only the integration layer (tool/agent connectivity) and leave execution semantics undefined: state and memory persistence, observability and auditability, resource governance, trust enforcement, and model mediation.

2. The Virtual Foundation Model (VFM) abstraction. An application-facing contract that decouples agent logic from physical foundation models (pFMs), providing stable semantics for checkpoint/resume, artifact persistence, hierarchical quotas, and principled escalation to verification — regardless of whether execution is workflow-shaped or loop-shaped.

3. A restructured virtualization criterion. Drawing on Popek and Goldberg's classical virtualization conditions (efficiency, resource control, fidelity), the authors argue that for FMs the third property should be replaced: not fidelity to underlying hardware, but progressive quality gain through self-evolution, achieved by learning which operations are "sensitive" and should trap. Self-evolution happens at the system level — updating prompts, policies, and structured memories — not by retraining model weights.

4. An FMOS architecture with three cooperating subsystems and a fast/slow execution path. A Data Agent (knowledge/context), a Composition Optimizer (model selection and routing), and a Trust & Reasoning Agent (verification and guardrails), plus mechanisms for observability, checkpoint/replay, and semantic debugging. Requests run on a lightweight fast path by default and escalate only when learned "traps" fire.

Main Findings

• The current stack is pre-operating-system. Because context management, tracing, tool execution, and verification are reimplemented per framework, improvements do not propagate and system-level optimization is unenforceable. The paper documents concrete divergence: Claude Code compacts long histories by summarizing and continuing; Cursor externalizes long tool outputs into files the agent can re-read; OpenCode auto-compacts near the context limit.

• Fragmentation spans five specific contracts. State and memory semantics (cross-session identity, replay/checkpointing); observability and auditability (provenance, decision paths); resource governance (budgets, quotas, multi-tenancy); trust enforcement (policy application, escalation, safe-by-default mediation); and model mediation (routing, caching, upgrades, rollbacks).

• Workflows and agentic loops lack a common substrate. Workflow pipelines assume typed steps and predictable logging; agent loops assume open-ended control flow, backtracking, and adaptive context growth. Encoded into incompatible runtimes, they compose only through fragile glue code — failing precisely where enterprise guarantees matter.

• Missing interface: context policy as a first-class object. Developers know which context is valuable and when, but cannot express that intent to the serving layer. Examples given include post-answer offload (web search agents), tool-output offload (enterprise infrastructure agents), adaptive skill unloading, and "retain thoughts, prune observations" (deep research agents). Each is an application-level policy over a system-level mechanism.

• Systems that gesture in this direction are incomplete. AIOS, MemGPT, and Llumnix explore scheduling, memory virtualization, and serving-level orchestration respectively, but the field lacks a single virtualization boundary that jointly governs knowledge, reasoning, verification, and trust.

• Self-evolution should be scaffolded by the system, not left to the model. The paper cites Evo-Memory and Evo-Test, which find that current models fail to reliably accumulate and reuse experience across tasks, and that reflection, memory augmentation, and reinforcement learning all struggle on test-time learning. FMOS supplies the signals (uncertainty, failure patterns, quality scores) while the control layer makes the evolution decisions and gates them with versioning, canaries, and rollback.

• The authors pre-empt five counterarguments. (a) Models will become good enough: longer inference-time reasoning cannot gather new evidence or adapt in dynamic environments without interaction. (b) Frameworks will encompass everything: frameworks wire components but provide no system-layer guarantees; multi-agent workflows still break on validation, context loss, rollback, and coordination. (c) MCP/A2A suffice: they intentionally stop short of execution semantics and governance, producing compliant but brittle "bloat." (d) The OS analogy is misplaced since it sits above hardware: that is exactly the point — it virtualizes higher-level FM operations (knowledge, reasoning) that conventional OS abstractions cannot express. (e) A database would be better: a database is an external service an application must explicitly query, whereas FMOS intervenes transparently beneath the model interface.

• Two case studies illustrate the abstractions. In scientific discovery, a query like "assess catalytic activity for hydrogen evolution of this MoS₂ microscopy tile" triggers a knowledge trap retrieving image regions plus supporting literature, routes to a domain- and vision-capable model, and escalates to validation against scientific priors. In enterprise technical support, a hierarchical decision tree of past interactions lets the agent zoom between the current troubleshooting node and the broader path, paging context as needed rather than managing a fixed window manually.

Methodology in Plain English

This is a position paper: it presents no experiments, datasets, or empirical results. The method is argumentation by analogy and architectural proposal.

The authors begin from an observed engineering pain point — agent frameworks each carry private, incompatible runtimes — and then reason by analogy to the history of operating systems, where programs once reimplemented their own I/O, memory, and scheduling until a shared layer absorbed those cross-cutting concerns. They borrow a classical result from computer architecture (Popek and Goldberg's conditions for virtualizable instruction sets) and adapt it: efficiency and resource control carry over to context windows and reasoning operations, but fidelity is replaced by managed self-evolution, because the goal is progressive quality gain rather than exact equivalence.

Because foundation models have no fixed instruction set, the authors argue that what counts as a "privileged" or "sensitive" operation must be learned — offline or in context — from interaction traces, and they propose an "FM trap" mechanism analogous to how MCP enables tool calling for models trained with function-calling ability. From there they sketch an architecture (three subsystems behind one interface, a fast path with escalation traps), walk through two application scenarios, and rebut alternatives before closing with a community call to action.

Why This Matters

Impact on research: The paper reframes an engineering mess as a systems-research agenda. It proposes a new named abstraction (the VFM) and a virtualization boundary as objects of study, and it shifts the locus of learning from model weights to system policies — prompts, routing rules, memory schemas, verification depth — which are auditable and rollback-able in ways that parametric updates are not. If adopted, it would open research directions in trap learning, policy-to-mechanism mapping, and system-level evaluation metrics.

Real-world applications:

• Enterprise technical support and IT operations — agents that must hold a detailed view of the current decision node while retaining the broader troubleshooting path, with auditable escalation for sensitive actions.
• Scientific discovery — multi-agent systems spanning hypothesis generation, simulation, and analysis, where evidence is multimodal (tables, figures, time series) and outputs must respect physical constraints.
• Personal AI assistants over local files (Moltbot, Agent Zero, Claude Cowork) — the paper names these as natural FMOS testbeds because they require sandboxing, durable memory, and policy-governed action mediation.
• Multi-tenant model serving — routing, caching, and sharing across a pool of models under explicit latency, cost, and quota constraints.

Industry relevance: The paper is authored largely by Hewlett Packard Enterprise researchers with a University of Chicago/Argonne co-author, and it is explicitly addressed in part to platform builders and standards bodies (it cites the Linux Foundation's Agentic AI Foundation). The practical stakes are portability of agent behavior across harnesses, propagating improvements across applications without rewrites, and turning safety and compliance from per-application add-ons into first-class, externally controllable governance mechanisms. The economic argument is joint optimization: coordinating retrieval, model choice, quotas, and verification depth as one problem yields efficiencies fragmented stacks cannot reach.

Future Directions

1. Define and standardize the core abstractions. The ML and systems communities must converge on a minimal set of system-layer contracts — the VFM lifecycle, isolation semantics, and guarantees — including standardized interfaces for context management, knowledge augmentation, reasoning control, and trust enforcement.

2. Build open reference architectures and prototypes. Platform builders should produce modular FMOS implementations that intercept execution through existing interfaces (OpenAI-compatible APIs, MCP endpoints, framework hooks) without application rewrites, and that coexist with popular agent frameworks.

3. Establish system-level benchmarks. Evaluation must move beyond task accuracy to measure context efficiency and knowledge reuse, robustness under evolving policies and data, cost–quality trade-offs from dynamic routing and reasoning escalation, reproducibility and auditability under FMOS mediation, and longitudinal self-evolution.

4. Align protocols and governance, and build long-lived testbeds. Standards bodies and enterprises should define contracts for safety enforcement, privilege levels, and external administrative control; the community should maintain persistent, evolving-agent testbeds and open repositories of reusable components (data agents, trust agents, model evaluators).

Open questions the paper leaves unresolved: how "sensitive" FM operations are actually learned and specified given the absence of a fixed instruction set; how much of the virtualization boundary can be enforced without model cooperation; and how to measure and verify progressive quality gain over time.

Target Audience

AI systems and infrastructure researchers; platform and serving engineers building agent runtimes or model-serving stacks; ML evaluation researchers interested in system-level rather than task-level benchmarks; enterprise architects and governance/compliance leads adopting agentic AI at scale; and standards-body participants working on agent interoperability protocols. Readers looking for experimental results or a concrete implementation should look elsewhere — this paper is a roadmap and an argument, not a demonstrated system.

Authors’ abstract

AI applications have shifted from single, monolithic foundation models (FM) to compound agentic systems. Yet today's stacks remain fragmented: even as protocols (e.g., MCP, A2A) ease tool/agent connectivity, each framework embeds an implicit runtime for state, memory, budgets, and guardrails, making behavior non-portable and governance brittle. It mirrors computing before operating systems, when every program re-implemented basic services. This position paper argues that the field now needs a Foundation Model Operating System (FMOS) -- a system layer that virtualizes FM interactions analogous to how virtual machines abstract physical hardware, giving applications the illusion of dedicated, trustworthy FM instances with effectively unbounded capabilities. Internally, the FMOS orchestrates knowledge across memory tiers, model selection and resource allocation, and verification and policy enforcement. Like the human brain switching between fast intuition and slow deliberation, the FMOS learns when to intervene and when to let inference proceed directly and continuously adapting its policies based on operational experience.