Skip to content
AI.info

The Pulse

Together AI Describes Polymorphic Inference for Serving Open Models as Traffic and Hardware Change

Together AI describes Polymorphic Inference as an approach that adapts to changing models, workloads and hardware. The company says Together Compiler is one of its first projects in this research agenda and is already running on some production inference workloads.

Together AI Describes Polymorphic Inference for Serving Open Models as Traffic and Hardware Change

Polymorphic Inference responds to changing models, traffic and hardware

On November 4, 2026, Together AI described Polymorphic Inference as an approach to serving open models that adapts as models, traffic and hardware change. The blog presents the shift from Static Inference to Polymorphic Inference as a response to faster model releases, more varied architectures and shifting production workloads. 80% of traffic on OpenRouter goes to models released in the last three months, Together AI says.

Together AI defines Static Inference as inference involving closed models, traditional Transformer architectures, single users and a single hardware platform. A typical vLLM, SGLang or InferenceX / AgentX recipe launches with a static, hyperparameter-tuned inference engine configuration, according to Together AI, which adds that a configuration optimized for one setting may not be the best one for another. Open-source kernels often assume static traffic shapes that can be miscalibrated against real production traffic, the blog also says.

Model architectures, traffic shapes and hardware are changing

All of the top-ten most popular OSS models in use today use different architectures, according to Together AI. Traffic can shift when a product or algorithmic change affects a harness, when user behavior changes, or when post-training changes traffic shapes, including how many tokens a model spends thinking. Hardware is changing too, according to Together AI, which cites inference-serving changes from B200 to GB300 and says it is on the cusp of seeing the first Vera Rubin deployments.

Every variation in model architecture, including MoE, and workload, including dynamic workloads larger than batch size 1, requires different technical approaches, Together AI says. Together AI points to an earlier design choice as a lesson: megakernel fusion is not the right design goal for modern Polymorphic Inference. Experience putting some of the first megakernels into production for customers showed that variations in architecture and workload call for different approaches, the blog says.

Together Compiler recasts inference optimization as an agentic search problem

Together Compiler is one of the projects Together AI is announcing as a first step toward its goals for Polymorphic Inference. The company describes it as a rapid AI-powered performance optimization RSI loop. Instead of treating inference optimization as a conventional compiler translation task, Together Compiler frames it as an agentic search problem. It identifies a core set of key optimization patterns and pairs a swarm of agents with a high-fidelity persistent engine for fast iterations.

The system is running in production today on some of Together AI’s most popular serverless endpoints, where it has found further gains on top of already highly optimized code. Together Compiler powers Continuous Compilation optimization loops across many of the company’s top production inference workloads. The work was the result of many months of work by Together’s Frontier Performance team, notably Reyna Abhyankar and Dylan Lim.

Together Compiler is available to end customers in limited beta today

Together Compiler is available to end customers in limited beta today. Customers will soon be able to use it to optimize dedicated endpoints for their unique workloads at the click of a button. Performance optimization needs to account for the workload, including the model, engine, traffic shape and customer use case, the company says.

Shape-shifting speculative decoding and ThunderAgent Router

Speculative decoding needs to adapt per-domain, per-request, per-token and as a function of traffic load, Together AI says. Inference engine configurations proposed in open-source repositories, including VLLM, InferenceX and AgentX, launch with a fixed speculative decoding draft model and lookahead parameter, according to Together AI.

For routing, Together AI released ThunderAgent as an ICML 2026 Spotlight paper and has expanded on these foundations in production. The blog describes the need to dynamically route requests across workers and deployment configurations across clusters because serverless inference users bring differing traffic shapes and service-level agreements.

Sources

  1. Original source · Blog post Together AI together.ai

Companies in this story

Related stories

Explore

More articles