sgl-project/sglang · error · ValueError

MXFP8 fused prologue requires K/V scale buffers.

Error message

MXFP8 fused prologue requires K/V scale buffers.

What it means

In the MXFP8 path of inkling_attn_prologue_verify, K and V must be written into scale-factor buffers (sfk, sfv) that live next to the paged KV cache — the kernel writes interleaved MXFP8 scales itself, so the caller must provide both buffers. The check sfk is None or sfv is None fires when the attention backend did not allocate or pass the MXFP8 scale caches, indicating the memory pool lacks the scale-factor tiers.

Source

Thrown at python/sglang/kernels/ops/attention/inkling_attn_prologue.py:94

    draft_token_num: int,
    activation: str | None = None,
    use_residual: bool = True,
    do_store: bool = True,
    mxfp8_quant: bool = False,
    sfk: torch.Tensor | None = None,
    sfv: torch.Tensor | None = None,
    page_size: int = 128,
    log_scaling_tau: torch.Tensor | None = None,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:
    """Returns fresh contiguous (q_normed, k_normed, v_conv) [T, dq/dkv];
    KV rows are also scattered into k_buf/v_buf at ``loc`` (the attention call
    should pass save_kv_cache=False)."""
    t = qkvr.shape[0]
    if mxfp8_quant:
        if dq % 128 != 0 or dkv % 128 != 0:
            raise ValueError("MXFP8 fused prologue requires head_dim-aligned Q/K/V.")
        if sfk is None or sfv is None:
            raise ValueError("MXFP8 fused prologue requires K/V scale buffers.")
        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)
        if sfk.shape != sf_shape or sfv.shape != sf_shape:
            raise ValueError(
                "MXFP8 fused prologue requires interleaved K/V scale buffers "
                f"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}."
            )
        if not sfk.is_contiguous() or not sfv.is_contiguous():
            raise ValueError(
                "MXFP8 fused prologue requires contiguous interleaved SFK/SFV."
            )
        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)
        sfq_u8 = torch.empty(
            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device
        )
        sfk_u8 = sfk.view(torch.uint8)
        sfv_u8 = sfv.view(torch.uint8)
    else:
        q_out = torch.empty(t, dq, dtype=qkvr.dtype, device=qkvr.device)

View on GitHub (pinned to 0132848349)

Solutions

1. Allocate/provide sfk and sfv scale buffers with shape [k_buf.shape[0]//page_size, dkv//128, 32, page_size//32, 4]
2. Ensure the KV cache pool is created in MXFP8 mode so scale tiers exist and are passed to the prologue
3. If MXFP8 is not intended, pass mxfp8_quant=False

Example fix

# before
q, k, v = inkling_attn_prologue_verify(qkvr, k_buf, v_buf, loc, ..., mxfp8_quant=True)  # sfk/sfv omitted
# after
q, k, v = inkling_attn_prologue_verify(qkvr, k_buf, v_buf, loc, ..., mxfp8_quant=True, sfk=sfk, sfv=sfv)

if mxfp8_quant:\n    assert sfk is not None and sfv is not None, 'MXFP8 requires scale buffers'

When it happens

Trigger: Calling inkling_attn_prologue_verify with mxfp8_quant=True but sfk=None or sfv=None — typically because the KV cache pool was created without MXFP8 scale buffers or the caller forgot to fetch them from the token allocator.

Common situations: Enabling MXFP8 KV-cache quantization on a server whose memory pool was built with a non-MXFP8 layout; refactoring the pool so scale buffers became optional and the fallback path silently passes None.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• MXFP8 fused prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused prologue requires interleaved K/V scale buffers
• MXFP8 fused prologue requires contiguous interleaved SFK/SFV
• MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused decode prologue requires K/V scale buffers.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3a0544e1eca39fa1. Report an issue: GitHub.