Linkerd Maker Buoyant Releases Kinference, an Open Source Control Plane for AI Inference

The project is aimed at companies that run open-weight models on their own infrastructure and serve them to many internal teams. Those companies already have tools for serving a single model or getting more out of a GPU. Buoyant chief executive William Morgan wrote in a launch post dated November 4 that the company built Kinference because customers were struggling to turn those pieces into a shared inference platform. Kinference adds a policy and accounting layer on top: per-tenant metrics, cache-aware load balancing and rules for which requests go to which models.

Kinference is licensed under Apache 2.0 and published on GitHub. It is still early: the current release is v0.1.0-alpha.7, according to MarkTechPost's review of the project. It is written in Rust and configured through five Kubernetes custom resources. Client applications point at a single HTTP endpoint and send traffic in OpenAI- or Anthropic-compatible formats.

The main feature is inference policies, which are declarative rules that route or reject requests based on the tenant, the model requested, the state of the GPU pool and a classification of the prompt. Examples on the project site include sending simple prompts to a cheaper model, sending overflow to an external cloud provider when local workers are queued up, keeping prompts flagged as containing personal data on local hardware, and running batch jobs only on lightly loaded GPUs. Teams can store policies in git and check them offline with a command-line tool before deploying them. Most rules pick whichever endpoint is predicted to return the first token fastest, a prediction Kinference bases on queue length and how much of the prompt is already cached on each worker.

In a benchmark Buoyant published alongside the launch, Kinference was tested against standard Kubernetes Service load balancing, which picks a pod at random. Each setup used three vLLM pods, each on one Nvidia L40S GPU, serving a Qwen 27-billion-parameter model on AWS. On a workload where prompts share long common prefixes, median time to first token fell from 6.88 seconds to 0.31 seconds, the roughly 95% cut that Buoyant leads with. On multi-turn conversations the median fell from 1.70 seconds to 0.59 seconds. On prompts with no shared content, median latency did not change in any meaningful way, although the slowest 5% of requests improved, which Buoyant attributes to Kinference taking queue length into account.

In capacity tests, Kinference handled about 7.4 times as many requests per minute within the latency target on the shared-prefix workload and about 3.5 times as many on multi-turn traffic. The standard setup never met the target at any concurrency level tested.

The benchmark has clear limits. Buoyant wrote and ran it for its own product, on one cluster with one model and older GPUs. The comparison was against random routing rather than other cache-aware routers such as the CNCF sandbox project llm-d or Nvidia's Dynamo. Buoyant also said its latency targets were set for offline agent workloads and are too slow for interactive chat.

Those rival projects currently support more inference engines and features such as splitting the prefill and decode stages across hardware, according to MarkTechPost's comparison. Kinference supports only vLLM for now, with SGLang and TensorRT-LLM listed as planned, and it uses Nvidia's dynamo-kv-router component to process cache events. Buoyant is positioning Kinference on governance features rather than raw speed: model aliasing, fallback to outside providers and per-tenant accounting. Per-tenant rate limits are listed as planned. The company has scheduled a livestream about the project for November 19.