Cornfigurator: Automated Planning for Any-to-Any Multimodal Model Serving

AI summary

Overview

Research area: Machine learning systems — specifically automated deployment planning and inference serving for multimodal "Any-to-Any" (A2A) models.

Technical level: Advanced. The paper assumes familiarity with LLM serving runtimes, colocation vs. disaggregation, tensor/expert/sequence parallelism, batch sizing, goodput, and latency CDFs.

Scope: The paper presents Cornfigurator, the first automated deployment planner for generic Any-to-Any multimodal model inference serving, which explores colocation, disaggregation, and mixed strategies to maximize per-request-type goodput.

What This Paper Is About

Any-to-Any multimodal models accept combinations of text, image, video, and audio as input and can generate them as output, so different requests traverse different paths through a graph of heterogeneous components. Deploying such models well requires deciding which components to colocate or disaggregate and how to configure each executor, but existing systems either demand manual expert tuning or only handle special cases such as Multimodal LLMs. Cornfigurator automates this planning by reasoning about each request type separately and searching the full space of deployment strategies to maximize the throughput of requests that meet their latency targets.

Key Contributions

1. Identification of a gap: The authors identify the absence of automated deployment planning for generic Any-to-Any model serving, where existing mechanisms either require manual expert effort or only generalize to special cases like MLLMs.
2. The Cornfigurator planner: An automated planner for generic A2A models that reasons about each request type individually, navigating graph-level (colocation/disaggregation) and executor-level (parallelism degree, batch size) decisions along with GPU allocation and routing probabilities.
3. A coarse-to-fine evaluation pipeline: Plan evaluation combines network-flow throughput estimation, Monte Carlo latency sampling, and a request-level simulator, with exact pruning after each phase to reduce a large candidate set to a final plan.
4. Empirical evaluation on recent A2A models: Plans generated by Cornfigurator "either match or deliver 1.12×–6.32× higher goodput" compared to plans used by existing systems and expert-tuned deployment plans, demonstrated on Qwen 3 Omni, Qwen 3 VL, InternVL 3, and Qwen Image. The system is released as open source at https://github.com/cornserve-ai/cornfigurator.

Main Findings

• Heterogeneity within a single model is extreme: In Qwen 3 Omni, the thinker LLM achieves nearly 30× higher request throughput than the talker LLM on A100. The paper also reports that the throughput difference between audio input and audio output components is nearly 200×.
• Per-component throughput spans orders of magnitude: Table 2 reports per-component throughput (requests/s) on A100-80GB. For Qwen 3 Omni: image input 5.43, video input 2.93, audio input 21.43, text in/out 2.15, audio output 0.12. Qwen 2.5 Omni: 15.64, 1.28, 34.04, 1.09, 0.28. Qwen 3 VL: 8.95, 0.97, 1.56. Qwen 2.5 VL: 12.04, 0.89, 1.63. InternVL 3: 1.13, 0.74, 0.59. Qwen-Image: 15.67, 0.20.
• No silver-bullet deployment strategy exists: Disaggregation lets components scale independently but consumes GPUs that could otherwise hold LLM KV cache; higher parallelism reduces per-request latency but may hurt throughput through communication overhead; larger batch size improves throughput until GPU saturation but may hurt latency. The paper reports a heatmap for InternVL 3 38B across workloads on 8× A100-80GB GPUs showing the best-throughput and best-latency configurations vary by workload.
• The configuration space is very large: The paper describes the A2A deployment configuration space as being on the order of millions of configurations.
• Single global latency targets are inadequate: With a single global latency constraint, the constraint binds only on the heaviest request type (for example audio output), allowing the planner to freely degrade lighter types such as text. Per-type latency targets constrain each type according to its own requirements.
• Proportional latency budgets are principled: Appendix A proves that setting each type's latency target proportionally to its computation cost factors out scale differences between types, making the constraint equally tight for all types when their latency distributions have similar shapes.
• Goodput as the objective: Goodput for a request type is defined as the throughput of that type's requests meeting their latency target, so a plan with high raw throughput but violated latency targets is penalized, and a plan that cannot handle incoming load builds queues and drives goodput toward zero.
• Adaptation costs differ by change type: GPU budget changes and workload fraction changes can reuse profiling results; new request types or changed request characteristics require re-profiling the affected type; model architecture or hardware changes require profiling from scratch.
• The planner is implemented and reasonably sized: The planner is implemented in about 5K lines of Rust on top of Cornserve, and the paper reports that intermediate tensor transfer between disaggregated executors has a median latency of approximately 10 ms across representative tensor sizes and traffic rates.
• Detailed per-model evaluation results are not included in the provided content: The evaluation section as supplied is truncated at "We find," so per-model goodput figures, simulation accuracy results, and the plan-count/time breakdown referenced for Sections 6.5 through 6.8 are not reported here.

Methodology in Plain English

The authors treat an Any-to-Any model as a directed acyclic graph in which nodes are components (multimodal encoders, LLMs, generators) and edges are data dependencies. Each unique combination of input and output modalities is a "request type" that invokes its own subgraph.

Planning proceeds in stages. First, the planner enumerates simple logical subplans by walking subgraphs of the model and, for each pair of adjacent components connected by a colocatable edge, deciding whether to Keep them separate or Merge them into one node — producing everything from fully disaggregated to monolithic topologies. Second, it creates compound logical subplans by merging up to k_c simple subplans (default k_c = 2) that share nodes, with routing probabilities splitting traffic among the merged subplans. Third, subplans are composed into logical plans — supergraphs of up to k_s parallel paths (default k_s = 2) that collectively cover every request type, so that one path can specialize for text-output requests while another handles audio-output requests. Fourth, logical plans are turned into physical plans by assigning concrete executor counts and configurations to each node within the GPU budget N, ensuring at least one executor per node, and setting routing probabilities for each request type (discretized at 0.1, or 10%, by default).

Candidate physical plans are then evaluated coarse-to-fine. A network-flow estimate finds the bottleneck node and the aggregate request rate R_d that saturates it, and plans with redundant capacity are pruned. Monte Carlo sampling draws K requests from the workload, routes each through the plan, and builds per-type latency CDFs to produce per-type goodput estimates, after which Pareto-dominated plans are discarded. A request-level simulator then models the full pipeline, including queuing and inter-type interactions, at rate α·R_d, where α is a headroom factor; it produces throughput and latency CDFs per request type, and final selection maximizes aggregate goodput by default. Profiling feeds this pipeline: executors are deployed on target hardware and benchmarked under a sweep of configuration knobs at saturating request rate, measured only in the engine's steady-state window, with queuing delay subtracted to isolate pure processing time.

Why This Matters

Impact on research. The paper reframes multimodal serving as a per-request-type optimization problem rather than an architecture-prescribed one, and it argues that existing disaggregation systems (vLLM-Omni, SGLang-Omni) provide mechanisms without a planner while special-case systems (ModServe, EPD) do not generalize. It also introduces a search formulation that composes specialized subplans, an idea applicable to other serving problems with heterogeneous request populations.

Real-world applications:

• Serving multimodal assistants that mix lightweight text responses with heavier audio or image responses, where the two have different latency expectations.
• Deploying MLLMs such as Qwen 3 VL or InternVL 3 that accept text, image, and video input but return text.
• Operating text-to-image models such as Qwen Image or GLM Image, where generation is far slower than text processing.
• Running omni models such as Qwen 3 Omni that combine image, video, and audio encoders with thinker and talker LLMs and a vocoder.

Industry relevance. The reported 1.12×–6.32× goodput improvement over existing and expert-tuned plans translates directly into GPU cost and serving capacity. Because the planner is runtime-agnostic and was demonstrated on Cornserve, it can in principle be combined with a range of runtimes, and its reuse of profiling results for GPU-budget and workload-distribution changes keeps re-planning cheap.

Future Directions

• Broadening the runtime evaluation: The paper demonstrates Cornfigurator on Cornserve; how well its plans transfer to other runtimes with different executor types and configuration knobs is an open question the paper frames as future applicability of its runtime-agnostic design.
• Scaling the search limits: The merge limit k_c and composition limit k_s bound the search explosion and default to 2. The paper varies these in Section 6.7, but the tradeoff between planner runtime and plan quality at higher limits remains a question the authors leave to that analysis.
• Finer routing granularity: Routing probabilities default to a 0.1 step size, with finer granularities supported at the cost of more physical plans to evaluate. Whether the extra accuracy justifies the extra search is a parameter the planner leaves tunable.
• Reducing the cost of model and hardware changes: Migration to a new GPU type or a changed model architecture requires re-running profiling from scratch, which the authors call the most expensive adaptation. Automating or amortizing that re-profiling is an evident next step.

Target Audience

This paper is most valuable to ML systems researchers and engineers who build or operate inference serving infrastructure for multimodal models, particularly those working on disaggregated serving, LLM/MLLM deployment, and auto-tuning of parallelism and batching. It also suits practitioners with GPU fleets who must decide how to allocate devices across encoders, LLMs, and generators under latency SLOs, and researchers interested in applying combinatorial planning and simulation-based search to serving problems. Readers without background in serving runtimes, goodput, and parallelism strategies will find the material advanced.

Authors’ abstract

Any-to-Any models are an emerging class of multimodal models that accept combinations of text and multimodal data as input and generate them as output, introducing heterogeneous computation paths and component scaling characteristics. There are existing mechanisms for deploying Any-to-Any models--or special cases of them--for inference serving, but they either require manual effort and expertise to tune, or do not generalize to generic Any-to-Any models. We present Cornfigurator, the first deployment planner for generic Any-to-Any model inference serving. The goal of Cornfigurator is to maximize the overall goodput of serving the model, defined as the throughput of requests meeting their latency targets. To do so, based on model and workload characteristics, Cornfigurator explores the full spectrum of deployment strategies, from colocation to disaggregation and mixing different strategies. Cornfigurator performs coarse-to-fine statistical evaluation to efficiently navigate the large space of candidate plans. Plans generated by Cornfigurator either match or deliver 1.12$\times$-6.32$\times$ higher goodput compared to existing systems and expert-tuned deployment plans.