sgl-project/sglang · error · ValueError

MXFP8 fused decode prologue requires K/V scale buffers.

Error message

MXFP8 fused decode prologue requires K/V scale buffers.

What it means

The decode prologue in MXFP8 mode writes interleaved K/V scale factors as it scatter-stores KV rows, so the caller must pass both sfk and sfv scale buffers. The check sfk is None or sfv is None fires when quantization is on but the scale caches were not provided — typically because the memory pool lacks the MXFP8 scale tiers or the caller did not fetch them.

Source

Thrown at python/sglang/kernels/ops/attention/inkling_attn_prologue.py:327

    mxfp8_quant: bool = False,
    sfk: torch.Tensor | None = None,
    sfv: torch.Tensor | None = None,
    page_size: int = 128,
    log_scaling_tau: torch.Tensor | None = None,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:
    """Decode {k/v decode-conv + conv-cache shift-update (+track) + qk-norm
    (+ KV store)} in one kernel. Returns fresh (q_normed, k_normed, v_conv).
    The k/v conv caches are shift-updated in place (fused_decode_update
    semantics). With ``do_store`` the KV rows are scattered into k_buf/v_buf at
    ``loc``; MXFP8 mode also quantizes Q and writes interleaved K/V scales."""
    t = qkvr.shape[0]
    if mxfp8_quant:
        if dq % 128 != 0 or dkv % 128 != 0:
            raise ValueError(
                "MXFP8 fused decode prologue requires head_dim-aligned Q/K/V."
            )
        if sfk is None or sfv is None:
            raise ValueError("MXFP8 fused decode prologue requires K/V scale buffers.")
        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)
        if sfk.shape != sf_shape or sfv.shape != sf_shape:
            raise ValueError(
                "MXFP8 fused decode prologue requires interleaved K/V scale buffers "
                f"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}."
            )
        if not sfk.is_contiguous() or not sfv.is_contiguous():
            raise ValueError(
                "MXFP8 fused decode prologue requires contiguous interleaved SFK/SFV."
            )
        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)
        sfq_u8 = torch.empty(
            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device
        )
        sfk_u8 = sfk.view(torch.uint8)
        sfv_u8 = sfv.view(torch.uint8)
    else:
        q_out = torch.empty(t, dq, dtype=qkvr.dtype, device=qkvr.device)

View on GitHub (pinned to 0132848349)

Solutions

  1. Allocate and pass sfk/sfv with shape (k_buf.shape[0]//page_size, dkv//128, 32, page_size//32, 4)
  2. Create the KV pool with MXFP8 support so scale buffers exist
  3. Fall back to mxfp8_quant=False if quantized decode is not required

Example fix

# before
q, k, v = inkling_attn_prologue_decode(qkvr, k_buf, v_buf, loc, ..., mxfp8_quant=True)
# after
q, k, v = inkling_attn_prologue_decode(qkvr, k_buf, v_buf, loc, ..., mxfp8_quant=True, sfk=sfk, sfv=sfv)
Defensive patterns

Strategy: validation

Validate before calling

if mxfp8_quant:\n    assert sfk is not None and sfv is not None

Prevention

When it happens

Trigger: Calling inkling_attn_prologue_decode with mxfp8_quant=True and sfk=None or sfv=None, e.g. a backend that allocated k_buf/v_buf only, or a code path that dropped the scale buffers when threading arguments.

Common situations: Decode batches after enabling MXFP8 with a pool allocated without scale tiers; partial refactors where the decode path was not updated to pass sfk/sfv like the prefill path was.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e2f10eb8bbb89aea. Report an issue: GitHub.