The Pulse
Together AI Introduces Compiler With Up to 2.6x Higher Inference Throughput
Together AI introduced Together Compiler, a system that uses agents and persistent inference engines to optimize for changing workloads. The company reports up to 2.6x higher throughput across several open models and says the compiler is al
AI.info Team · Written with AI · fact-checked ·
Together AI published its introduction to Together Compiler on Nov. 4, describing a system that continuously tunes inference for changing workloads. The compiler, called TKCC, uses agents to generate optimizations based on traffic shape, model and hardware. The company says it is already serving trillions of tokens.
Persistent engines keep experiments moving
Together AI frames TKCC as a way to optimize across the many combinations of models, hardware and customer traffic that inference providers encounter. Its post says the top 10 open-source large language models on OpenRouter do not share an architecture. The company also says it serves many trillions of tokens per day across thousands of customers, whose workloads differ in batch size, sequence length, expert imbalance, attention sparsity and session duration.
TKCC’s main mechanism is a persistent inference engine that starts once and hot-reloads candidate forward passes without restarting. Together AI says conventional agentic optimization systems often relaunch engines such as vLLM, SGLang or TRT-LLM for each experiment, and that engine startup can take tens of minutes. In the company’s table, total time to serving is 467 seconds for the status quo and 9.0 seconds for Together Compiler.
Multiple agents generate optimization ideas in parallel and communicate through a message board, according to the post. TKCC also uses two types of persistent engines: fast engines load a few layers for rapid iteration and low-variance timings, while slower engines load more layers to test optimizations against real data distributions. Together AI says the design aims to balance experiment speed with measurement fidelity.
Traffic changes the optimization target
The post describes three sources of workload variation: regional patterns, new model releases and bursts of customer requests. For one country’s working hours, Together AI has observed prompts 1.4 times longer than in another country’s working hours, at 97k versus 70k. The company says prompt length increases by 80% two weeks after a model release as users move from chat-based evaluation toward agentic workloads.
Bursts can suddenly double prompt lengths and lower cache hit rates, the post says. Together AI argues that each new deployment needs specialized optimizations as user behavior changes. Its compiler takes the traffic shape alongside the model and hardware as inputs.
A cost model prunes configuration choices
TKCC uses a custom cost model to select parallelism strategies, including TP, EP and DCP. Together AI says measuring every possible configuration is impractical, while choosing from hardware specifications or microbenchmarks alone can miss traffic details such as uneven prompt lengths and MoE expert imbalance. The company’s model combines hardware specifications, microbenchmarks and measured forward passes to narrow the configuration search.
The compiler’s measurement process also uses fast engines with a few loaded layers and slower engines with more layers. Together AI says input data distributions can change which optimization strategies perform best, particularly for models with mixture-of-experts and sparse-attention operations. The company presents testing against real data distributions as a way to check whether optimizations hold beyond rapid experiments.
Benchmark gains span models and workloads
Together AI reports that TKCC achieves up to 2.6x higher tokens per second across Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash in less than 24 hours. On SemiAnalysis’ InferenceX low-latency benchmark, the company reports 1,111 TPS on MiniMax M3.1-Flash and 1,205 TPS on DeepSeek v4.1-Flash. For those results, Together AI says its cost model selected the parallelism configuration instead of the InferenceX configuration.
On the Llama 1B batch-size-1 workload, Together AI reports 1.7x higher TPS than open-source megakernels. The post says TKCC sets the performance frontier across the tested InferenceX AgentX settings, outperforming optimized baselines by up to 2.6x higher TPS. Together AI also reports that the system generalizes to weather modeling, partial differential equations, time-series work and biology, and describes its use as a general-purpose AI compiler beyond language-model inference.