Zyphra Releases Open FP8 Attention Kernels That Outrun AMD's Own on MI355X Chips

On activations captured from real models, ZYFA's 8-bit floating point (FP8) kernels ran 1.16 to 1.76 times faster than AITER, AMD's default kernel library, according to Zyphra's technical paper. The gains were 1.16 times on Mistral-7B, 1.17 times on Qwen3-8B and 1.76 times on DeepSeek-V2-Lite. At the common head dimension of 128, Zyphra said ZYFA's FP8 error was 62% to 78% lower than AITER's. The code is released under the Apache 2.0 licence.

The release matters for anyone weighing AMD chips against Nvidia's for AI inference. Hardware performance depends heavily on low-level software, and a third party extracting substantially more from AMD's flagship accelerator than AMD's own libraries suggests buyers may be leaving performance unused on the platform.

Zyphra's paper argues that FP8 attention on the MI355X is limited by the softmax step rather than by matrix multiplication. FP8 matrix instructions run at twice the speed of 16-bit ones, but the softmax calculation between the two matrix products runs on vector units at the same speed in either precision. Its traces showed AITER's matrix units sitting idle for 31.7% of cycles. ZYFA replaces the exponential in softmax with a single multiply-add, a method Zyphra calls Lean P, and alternates two groups of threads so one performs matrix work while the other computes probabilities. Zyphra said this keeps the matrix units busy 86% of the time.

On synthetic benchmarks at 32,000 tokens, the FP8 kernel reached 2,672 teraflops per second, or 53% of the chip's datasheet peak, against 1,944 for AITER as measured by Zyphra. The company said real-model gains are smaller partly because the MI355X lowers its clock speed when more data bits change, and recommended benchmarking on real activations.

The kernels cover only the forward pass of causal attention during prefill, on the MI355X alone. They do not yet support decoding or paged key-value caches, and Zyphra did not test Nvidia hardware. ZYFA powers the attention engine of Zyphra Cloud, the company's AMD-based inference service, which launched earlier this year.