Supermicro, IBM and Nvidia Launch Shared KV Cache Storage to Speed Up AI Inference

Supermicro announced the product in a press release on November 5. It combines Supermicro's Petascale all-flash storage servers, which run on AMD EPYC 9005 Series processors, with IBM Storage Scale software, Nvidia's Dynamo inference framework and Nvidia Spectrum-X Ethernet networking.

The system targets a cost that is rising quickly for anyone serving large language models. Before a model answers a prompt, it computes key-value states for every token in the context, a step known as prefill. These states, called the KV cache, normally sit in GPU or CPU memory and get evicted when space runs out. Agentic workflows, coding assistants, long-context retrieval and multi-turn chat can all produce more KV data than that memory holds. The GPUs then have to repeat the work, which delays the first token of each response.

Keeping the cache on a shared storage tier means any GPU node in a cluster can pick up context that another node has already computed. The companies say this raises GPU utilisation, throughput and the number of users a cluster can serve at once, without adding GPUs in proportion.

Supermicro said joint testing with Nvidia on a 120-billion-parameter model showed time to first token up to 56 times faster at prompts of 130,000 tokens. As prompts grew from 30,000 to 130,000 tokens, time to first token rose by 0.377 seconds with the persistent cache and by 29.55 seconds without it, the company said. The tests ran at cluster level and included scenarios with high concurrency and noisy-neighbour network traffic.

The vendors ran the tests themselves. The full results appear in an IBM Redbook titled "Context without Limits: A High-Performance KV Cache Platform for Large-Scale AI Inference", which describes the same architecture. The release gave no pricing or general availability date, and it did not say which GPUs or how much storage were used in testing.

Charles Liang, Supermicro's president and chief executive, said enterprises and neoclouds need to remove bottlenecks to get more out of their GPUs as inference workloads grow. "Our solution with IBM and NVIDIA clearly demonstrates these performance benefits," he said.

Sam Werner, general manager of IBM Storage, called the collaboration "an important step toward making enterprise AI inference more scalable". He described the system as a way to scale inference capacity separately from compute. Shruti Koparkar, Nvidia's AI inference product lead, said it "creates a high-performance shared storage tier that can reduce redundant computation".

Storage suppliers have spent the year building around this problem. Nvidia has been standardising ways to move KV cache off GPUs and onto NVMe storage, and its Dynamo software manages where the cache sits. For buyers, the result is a choice between adding GPU memory and buying a storage layer to hold context that has already been computed.

Supermicro said it will deliver the system with factory rack integration and on-site deployment services, which it says can cut deployment time from weeks to days.