Cornserve: A Distributed Serving System for Any-to-Any Multimodal Models

AI summary

Overview

Research area: Distributed machine learning systems and inference serving infrastructure for multimodal AI models.

Technical level: Advanced (assumes familiarity with LLM serving, disaggregated inference, and distributed systems concepts, though the summary is written accessibly).

Scope: This paper introduces Cornserve, a distributed serving system that disaggregates arbitrary multimodal "Any-to-Any" models into independently scalable components and orchestrates their execution across GPUs and nodes.

What This Paper Is About

Modern multimodal models—called Any-to-Any models—can take text, images, video, or audio as input and produce any of those as output. Different requests take different paths through the model's internal computation graph, and each component (encoder, LLM, generator) scales very differently on hardware. The paper's goal is to build a general-purpose serving system that can disaggregate these models into separately scalable pieces and execute them efficiently, rather than forcing the whole model to run as one monolithic block on a single deployment.

Key Contributions

1. Flexible task abstraction: Three levels of abstraction—unit tasks (atomic components like encoders or LLMs), composite tasks (models expressed as Python code composing unit tasks), and apps (Python modules defining the entry-point serve function)—let developers express arbitrary Any-to-Any model logic in plain Python.

2. Model fission: The ability to split a model at arbitrary component boundaries into independently deployed, independently scaled components, each running on dedicated GPUs with specialized executors (Eric for encoders, a forked vLLM for LLMs, Geri for generators).

3. Record-and-replay distributed runtime: A two-phase execution mechanism that captures the actual subgraph of component invocations for a given request (record) and then replays it with real results (replay), supporting loops and branches without pre-compiling every request path.

4. Efficient data plane and component sharing: Sidecar daemons forward tensors producer-to-consumer via shared memory (intra-node) or RDMA/UCX (inter-node), bypassing the control plane; equivalent unit tasks across apps automatically share a single executor deployment to save GPUs.

Main Findings

• Throughput gains: Cornserve achieves 3.09x higher throughput on 8 GPUs and 3.81x on 16 GPUs for Qwen 2.5 Omni 7B versus a monolithic Hugging Face Transformers baseline.
• Enables previously unservable models: The monolithic baseline for Qwen 3 Omni 30B fails outright due to GPU out-of-memory; Cornserve's fission-based planner serves it successfully, scaling from 8 to 16 GPUs with a 2.68x throughput improvement (superlinear, because additional GPUs allow better component balancing).
• Tail latency reduction: For Qwen 2.5 Omni on 16 GPUs, P50 latency improves 3.24x, P95 improves 5.3x, and P99 improves 5.79x. Gains come from eliminating component interference (only one component runs at a time in monolithic deployment) and enabling continuous batching for the autoregressive audio generator.
• Independent scaling works in practice: The planner automatically replicates the audio generator 7 or 15 times (on 8 or 16 GPUs) to match its lower throughput relative to the LLM; for Qwen 3 Omni it assigns tensor parallelism to the thinker LLM while replicating talkers and generators.
• Forwarding overhead is minor: Sidecar tensor transfers add 5–10 ms (8 MB), 8–20 ms (16 MB), and 12–27 ms (32 MB) at 5–15 transfers per second—not the dominant latency factor since transfers only occur at fission boundaries.

Methodology in Plain English

The authors start from the observation that Any-to-Any models are really graphs of heterogeneous components—encoders, one or more LLMs, and generators—and that different requests walk different paths through that graph. Instead of hard-coding support for specific architectures, they let developers describe models in ordinary Python. A composite task's invoke method contains the model's logic, calling into unit tasks as needed.

The clever part is how they handle request-dependent control flow. Running the Python logic naively would require either knowing every request path in advance or paying round-trip costs per component. Instead, Cornserve first "records" execution: unit tasks return placeholder objects, and by matching object IDs across inputs and outputs, the system reconstructs the exact subgraph this request needs—done instantly, with no real computation. It then dispatches all unit task invocations to the appropriate executors simultaneously and "replays" the Python logic with the real results substituted in.

Deployment is managed on Kubernetes. A planner picks a layout based on component scaling characteristics, an allocator spawns Task Managers, and each Task Manager launches executors on its assigned GPUs. Between disaggregated components, per-GPU Sidecar daemons move tensors directly—shared memory within a node, RDMA across nodes. When two apps use identical unit tasks (say, the same vision encoder), Cornserve deploys only one copy.

Why This Matters

Research impact: The paper provides a general abstraction for serving any multimodal model with arbitrary computation structure, moving beyond the point solutions (vLLM for text, xDiT for diffusion) that dominate today. It formalizes the idea that disaggregation boundaries should be programmable rather than fixed to architectures like Prefill–Decode or Encode–Prefill–Decode.

Real-world applications:

• Voice assistants that take audio input and produce spoken responses with interleaved reasoning (Qwen Omni-style models).
• Image generation services where a text-only request bypasses an unused vision encoder entirely, saving compute.
• Video and audio generation pipelines where compute-heavy generators must be scaled independently of LLMs.
• Multi-model serving platforms where many models share encoders or other common components across tenants.

Industry relevance: Frontier multimodal serving is GPU-cost-dominated. Independently scaling components and sharing executors translates directly to lower cost per request and better hardware utilization. The Kubernetes-based, open-source implementation (23K lines of Python) makes it deployable in existing cloud infrastructure rather than requiring bespoke orchestration.

Future Directions

• Data-dependent control flow: The record-and-replay mechanism requires the composite task's control flow to be deterministic given the request. Handling branches driven by an LLM's own output—deciding mid-flight which task to invoke next—remains an open problem, currently punted to the app layer.
• Better planners: The planner currently relies on a learned cost model; improving its accuracy across more diverse component mixes and hardware topologies could yield further gains.
• Broader model coverage: Extending beyond the evaluated Qwen, InternVL, and Gemma families to diffusion transformers, video generation models, and hybrid architectures with different scaling laws.
• Reducing forwarding overhead: As models fragment into more components and cross-node transfers grow, optimizing the Sidecar data plane (or exploring topologies that keep hot component pairs co-located) becomes more important.

Target Audience

Systems researchers working on LLM and multimodal inference infrastructure; ML engineers deploying multimodal models in production; distributed systems practitioners interested in disaggregated serving and Kubernetes-based GPU orchestration; and graduate students studying the intersection of model architecture and serving efficiency. Readers without background in inference serving or distributed systems will find the paper dense but the architectural insights accessible.

Authors’ abstract

Any-to-Any models are an emerging class of multimodal models that accept combinations of multimodal data (e.g., text, image, video, audio) as input and generate them as output. Serving these models are challenging; different requests with different input and output modalities traverse different paths through the model computation graph, and each component of the model have different scaling characteristics. We present Cornserve, a distributed serving system for generic Any-to-Any models. Cornserve provides a flexible task abstraction for expressing Any-to-Any model computation graphs, enabling component disaggregation and independent scaling. The distributed runtime dispatches compute to the data plane via an efficient record-and-replay execution model that keeps track of data dependencies, and forwards tensor data between components directly from the producer to the consumer. Built on Kubernetes with approximately 23K new lines of Python, Cornserve supports diverse Any-to-Any models and delivers up to 3.81$\times$ higher throughput and 5.79$\times$ lower tail latency. Cornserve is open-source, and the demo video is available on YouTube.