The Pulse
Together AI Describes Polymorphic Inference for Serving Open Models as Traffic and Hardware Change
Together AI describes Polymorphic Inference as an approach that adapts to changing models, workloads and hardware. The company says Together Compiler is one of its first projects in this research agenda and is already running on some production inference workloads.

Polymorphic Inference responds to changing models, traffic and hardware
On November 4, 2026, Together AI described Polymorphic Inference as an approach to serving open models that adapts as models, traffic and hardware change. The blog presents the shift from Static Inference to Polymorphic Inference as a response to faster model releases, more varied architectures and shifting production workloads. 80% of traffic on OpenRouter goes to models released in the last three months, Together AI says.
Together AI defines Static Inference as inference involving closed models, traditional Transformer architectures, single users and a single hardware platform. A typical vLLM, SGLang or InferenceX / AgentX recipe launches with a static, hyperparameter-tuned inference engine configuration, according to Together AI, which adds that a configuration optimized for one setting may not be the best one for another. Open-source kernels often assume static traffic shapes that can be miscalibrated against real production traffic, the blog also says.
Model architectures, traffic shapes and hardware are changing
All of the top-ten most popular OSS models in use today use different architectures, according to Together AI. Traffic can shift when a product or algorithmic change affects a harness, when user behavior changes, or when post-training changes traffic shapes, including how many tokens a model spends thinking. Hardware is changing too, according to Together AI, which cites inference-serving changes from B200 to GB300 and says it is on the cusp of seeing the first Vera Rubin deployments.
Every variation in model architecture, including MoE, and workload, including dynamic workloads larger than batch size 1, requires different technical approaches, Together AI says. Together AI points to an earlier design choice as a lesson: megakernel fusion is not the right design goal for modern Polymorphic Inference. Experience putting some of the first megakernels into production for customers showed that variations in architecture and workload call for different approaches, the blog says.
Together Compiler recasts inference optimization as an agentic search problem
Together Compiler is one of the projects Together AI is announcing as a first step toward its goals for Polymorphic Inference. The company describes it as a rapid AI-powered performance optimization RSI loop. Instead of treating inference optimization as a conventional compiler translation task, Together Compiler frames it as an agentic search problem. It identifies a core set of key optimization patterns and pairs a swarm of agents with a high-fidelity persistent engine for fast iterations.
The system is running in production today on some of Together AI’s most popular serverless endpoints, where it has found further gains on top of already highly optimized code. Together Compiler powers Continuous Compilation optimization loops across many of the company’s top production inference workloads. The work was the result of many months of work by Together’s Frontier Performance team, notably Reyna Abhyankar and Dylan Lim.
Together Compiler is available to end customers in limited beta today
Together Compiler is available to end customers in limited beta today. Customers will soon be able to use it to optimize dedicated endpoints for their unique workloads at the click of a button. Performance optimization needs to account for the workload, including the model, engine, traffic shape and customer use case, the company says.
Shape-shifting speculative decoding and ThunderAgent Router
Speculative decoding needs to adapt per-domain, per-request, per-token and as a function of traffic load, Together AI says. Inference engine configurations proposed in open-source repositories, including VLLM, InferenceX and AgentX, launch with a fixed speculative decoding draft model and lookahead parameter, according to Together AI.
For routing, Together AI released ThunderAgent as an ICML 2026 Spotlight paper and has expanded on these foundations in production. The blog describes the need to dynamically route requests across workers and deployment configurations across clusters because serverless inference users bring differing traffic shapes and service-level agreements.
Sources
- Original source · Blog post Together AI together.ai
Companies in this story
Related stories
- Score Centering Keeps LLM RL Stable at 30% Under INT4 Stress
- GitHub Says New Git Storage Architecture Supported 5,000 Concurrent Agents in Internal Testing
- METR Estimates Anthropic’s Model Capabilities Are Advancing at About Twice Their Earlier Rate
- AMD Data Center Revenue Jumps 97% on Instinct and EPYC Sales
- Ballistic Ventures Seeks $490M for Two Cybersecurity Funds