The Pulse
Together AI Describes Polymorphic Inference for Serving Open Models as Traffic and Hardware Change
Together AI describes Polymorphic Inference as an approach that adapts to changing models, workloads and hardware. The company says Together Compiler is one of its first projects in this research agenda and is already running on some produc
AI.info Team · Written with AI · fact-checked ·
Polymorphic Inference responds to changing models, traffic and hardware
On November 4, 2026, Together AI described Polymorphic Inference as an approach to serving open models that adapts as models, traffic and hardware change. The blog presents the shift from Static Inference to Polymorphic Inference as a response to faster model releases, more varied architectures and shifting production workloads. Together AI says 80% of traffic on OpenRouter goes to models released in the last three months.
Together AI defines Static Inference as inference involving closed models, traditional Transformer architectures, single users and a single hardware platform. It separately says a typical vLLM, SGLang or InferenceX / AgentX recipe launches with a static, hyperparameter-tuned inference engine configuration, and that a configuration optimized for one setting may not be the best one for another. The blog also says open-source kernels often assume static traffic shapes that can be miscalibrated against real production traffic.
Model architectures, traffic shapes and hardware are changing
According to Together AI, all of the top-ten most popular OSS models in use today use different architectures. Traffic can shift when a product or algorithmic change affects a harness, when user behavior changes, or when post-training changes traffic shapes, including how many tokens a model spends thinking. Together AI says hardware is changing too, citing inference-serving changes from B200 to GB300 and saying it is on the cusp of seeing the first Vera Rubin deployments.
Together AI says every variation in model architecture, including MoE, and workload, including dynamic workloads larger than batch size 1, requires different technical approaches. It points to an earlier design choice as a lesson: megakernel fusion is not the right design goal for modern Polymorphic Inference. The blog says that experience putting some of the first megakernels into production for customers showed that variations in architecture and workload call for different approaches.
Together Compiler recasts inference optimization as an agentic search problem
Together Compiler is one of the projects Together AI is announcing as a first step toward its goals for Polymorphic Inference. The company describes it as a rapid AI-powered performance optimization RSI loop. Instead of treating inference optimization as a conventional compiler translation task, Together Compiler frames it as an agentic search problem. It identifies a core set of key optimization patterns and pairs a swarm of agents with a high-fidelity persistent engine for fast iterations.
The blog says the system is running in production today on some of Together AI’s most popular serverless endpoints, where it has found further gains on top of already highly optimized code. Together Compiler powers Continuous Compilation optimization loops across many of the company’s top production inference workloads. The work was the result of many months of work by Together’s Frontier Performance team, notably Reyna Abhyankar and Dylan Lim.
Together Compiler is available to end customers in limited beta today
Together Compiler is available to end customers in limited beta today. The blog says customers will soon be able to use it to optimize dedicated endpoints for their unique workloads at the click of a button. The company says performance optimization needs to account for the workload, including the model, engine, traffic shape and customer use case.
Shape-shifting speculative decoding and ThunderAgent Router
Together AI says speculative decoding needs to adapt per-domain, per-request, per-token and as a function of traffic load. It says inference engine configurations proposed in open-source repositories, including VLLM, InferenceX and AgentX, launch with a fixed speculative decoding draft model and lookahead parameter.
For routing, Together AI says it released ThunderAgent as an ICML 2026 Spotlight paper and has expanded on these foundations in production. The blog describes the need to dynamically route requests across workers and deployment configurations across clusters because serverless inference users bring differing traffic shapes and service-level agreements.