Together AI Says Agent-Driven Compiler Speeds Up Model Inference by Up to 2.6 Times

The idea behind the product is that a serving engine tuned once soon goes out of date, so the tuning should never stop. For inference providers, more tokens per second from the same chips means more users per GPU. That is the main way to bring down the cost of serving a model, and inference is a growing share of what companies spend on AI.

Together's own traffic data shows why it thinks one-off tuning falls short. It said prompts from users in one country averaged about 97,000 tokens, against 70,000 elsewhere, and that one customer's 90th-percentile input length grew from 18,000 to 33,000 tokens in three days. It also found that random test inputs activated 13% to 38% more mixture-of-experts components than real traffic did. One agent-written kernel took 20 microseconds on random data but up to 62 milliseconds on real data at a 70,000-token context.

How the Loop Works

The system takes a server launch command, a traffic profile and a hardware configuration. Agents then propose faster versions of the model's forward pass, a persistent engine measures each one, and the best code goes into production. When traffic drifts, the loop runs again.

Most agent-based optimisation setups restart the serving engine for every experiment. Together measured a cold start of 761 seconds for MiniMax M3.1 Flash on TensorRT-LLM. Its compiler starts the engine once and swaps in each candidate without a restart. A fast test tier runs 12 layers of the model for screening, and a slow tier runs 60 layers to confirm gains. The company said this two-tier setup tested 1,354 candidates against 168 for the slow tier alone and found a solution 7.2% faster. It put the overall gain in agent experiment throughput at 252 times an engine-restart baseline.

Before the agents start, a cost model ranks ways of splitting the model across chips. Together said that on Kimi K3 this found a production layout with up to 24% higher throughput.

The Benchmark Numbers

On the AgentX workload in SemiAnalysis's InferenceX benchmark, each setting ran for under 24 hours on no more than two nodes. At batch size 1, the gains varied widely. MiniMax M3.1 Flash on Nvidia B200 reached 1,111 tokens per second per user against a baseline of 633. DeepSeek v4.1 Flash on GB300 reached 1,205 against 943, and GLM 5.3 on B200 reached 417 against 329. Kimi K3 on GB300 moved from 423 to 450, and GLM 5.3 on AMD's MI355X gained only about 4%, from 463 to 480.

The system was tested on Nvidia B200, B300, GB300 and H100 chips, AMD's MI355X and AWS Trainium2. On the small Llama 1B model it beat Hazy Research's megakernels, Mirage and CUDA-graph baselines by up to 1.7 times. Together said it also sped up eight scientific models by between 1.5 and 6.4 times with bit-exact outputs. Anthropic's Claude Code, given the same task with its /goal feature, managed 1.2 to 4.4 times and failed on the AIFS-ENS 2.0 weather model.

One finding cuts against conventional practice. Agents given an expert-curated knowledge base produced 25% to 31% smaller speedups after 12 hours than agents simply told to make the forward pass "as fast as possible". After 10 trials, the ensemble scores were 6.2 times for the uninstructed agents and 4.9 times for the expert-guided ones. Custom logging structures took up as much as 43% of tool calls, and Together now gives its agents the engine's raw source code instead.

All the performance figures were reported by Together AI, and the InferenceX baselines it compared against date from October 27, 2026.