sgl-project/sglang · error · ValueError

MXFP8 fused prologue requires interleaved K/V scale buffers

Error message

MXFP8 fused prologue requires interleaved K/V scale buffers with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}.

What it means

MXFP8 scale buffers must follow the interleaved layout the Triton quantization kernel writes: sf_shape = (k_buf.shape[0]//page_size, dkv//128, 32, page_size//32, 4) — i.e. [pages, scale-blocks-per-page, 32, rows-per-block-of-32, 4 uint8 sub-scales]. The check compares both sfk.shape and sfv.shape against this exact tuple; any deviation (wrong page_size assumption, wrong dkv//128 count, or a flat scale buffer) is rejected with the expected vs got shapes in the message.

Source

Thrown at python/sglang/kernels/ops/attention/inkling_attn_prologue.py:97

    do_store: bool = True,
    mxfp8_quant: bool = False,
    sfk: torch.Tensor | None = None,
    sfv: torch.Tensor | None = None,
    page_size: int = 128,
    log_scaling_tau: torch.Tensor | None = None,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:
    """Returns fresh contiguous (q_normed, k_normed, v_conv) [T, dq/dkv];
    KV rows are also scattered into k_buf/v_buf at ``loc`` (the attention call
    should pass save_kv_cache=False)."""
    t = qkvr.shape[0]
    if mxfp8_quant:
        if dq % 128 != 0 or dkv % 128 != 0:
            raise ValueError("MXFP8 fused prologue requires head_dim-aligned Q/K/V.")
        if sfk is None or sfv is None:
            raise ValueError("MXFP8 fused prologue requires K/V scale buffers.")
        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)
        if sfk.shape != sf_shape or sfv.shape != sf_shape:
            raise ValueError(
                "MXFP8 fused prologue requires interleaved K/V scale buffers "
                f"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}."
            )
        if not sfk.is_contiguous() or not sfv.is_contiguous():
            raise ValueError(
                "MXFP8 fused prologue requires contiguous interleaved SFK/SFV."
            )
        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)
        sfq_u8 = torch.empty(
            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device
        )
        sfk_u8 = sfk.view(torch.uint8)
        sfv_u8 = sfv.view(torch.uint8)
    else:
        q_out = torch.empty(t, dq, dtype=qkvr.dtype, device=qkvr.device)
        sfq_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)
        sfk_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)
        sfv_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)

View on GitHub (pinned to 0132848349)

Solutions

1. Reallocate sfk/sfv exactly as torch.empty(k_buf.shape[0]//page_size, dkv//128, 32, page_size//32, 4, dtype=torch.uint8, device=...)
2. Confirm the page_size used to allocate matches the one passed to the prologue call
3. After upgrading sglang, re-derive the shape from the current formula instead of hardcoding it

Example fix

# before
sfk = torch.empty(num_pages, dkv//128, page_size, dtype=torch.uint8, device='cuda')
# after
sfk = torch.empty(k_buf.shape[0]//page_size, dkv//128, 32, page_size//32, 4, dtype=torch.uint8, device='cuda')

sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)
assert sfk.shape == sf_shape and sfv.shape == sf_shape, (sfk.shape, sfv.shape, sf_shape)

When it happens

Trigger: Calling inkling_attn_prologue_verify with mxfp8_quant=True where sfk/sfv were allocated with a different page_size, a different head grouping, or a legacy non-interleaved MXFP8 layout.

Common situations: Changing page_size in server args without reallocating scale buffers; version upgrades that changed the interleaved scale layout; reusing scale buffers allocated for dkv=128 on a dkv=256 layer.

Related errors

• MXFP8 fused prologue requires contiguous interleaved SFK/SFV
• MXFP8 fused decode prologue requires K/V scale buffers.
• MXFP8 fused decode prologue requires interleaved K/V scale b
• MXFP8 fused prologue requires K/V scale buffers.
• MXFP8 fused decode prologue requires contiguous interleaved

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/d8ee28981e9f5952. Report an issue: GitHub.