sgl-project/sglang · error · ValueError

MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.

Error message

MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.

What it means

The decode prologue (inkling_attn_prologue_decode) applies the same MXFP8 constraint as the verify/extend variants: with mxfp8_quant=True, dq and dkv must both be multiples of 128 to fit the MXFP8 microscaling block structure (128-element superblocks, 32-element scale groups). The decode-specific message distinguishes it from the prefill error so you know which path raised it.

Source

Thrown at python/sglang/kernels/ops/attention/inkling_attn_prologue.py:323

    use_residual: bool = True,
    track_mask: torch.Tensor | None = None,
    track_indices: torch.Tensor | None = None,
    do_store: bool = True,
    mxfp8_quant: bool = False,
    sfk: torch.Tensor | None = None,
    sfv: torch.Tensor | None = None,
    page_size: int = 128,
    log_scaling_tau: torch.Tensor | None = None,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:
    """Decode {k/v decode-conv + conv-cache shift-update (+track) + qk-norm
    (+ KV store)} in one kernel. Returns fresh (q_normed, k_normed, v_conv).
    The k/v conv caches are shift-updated in place (fused_decode_update
    semantics). With ``do_store`` the KV rows are scattered into k_buf/v_buf at
    ``loc``; MXFP8 mode also quantizes Q and writes interleaved K/V scales."""
    t = qkvr.shape[0]
    if mxfp8_quant:
        if dq % 128 != 0 or dkv % 128 != 0:
            raise ValueError(
                "MXFP8 fused decode prologue requires head_dim-aligned Q/K/V."
            )
        if sfk is None or sfv is None:
            raise ValueError("MXFP8 fused decode prologue requires K/V scale buffers.")
        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)
        if sfk.shape != sf_shape or sfv.shape != sf_shape:
            raise ValueError(
                "MXFP8 fused decode prologue requires interleaved K/V scale buffers "
                f"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}."
            )
        if not sfk.is_contiguous() or not sfv.is_contiguous():
            raise ValueError(
                "MXFP8 fused decode prologue requires contiguous interleaved SFK/SFV."
            )
        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)
        sfq_u8 = torch.empty(
            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device
        )

View on GitHub (pinned to 0132848349)

Solutions

Disable mxfp8_quant for this model/layer
Use 128-aligned head dims if MXFP8 is required
Gate the flag per layer: mxfp8_quant and dq % 128 == 0 and dkv % 128 == 0

Example fix

# before
q, k, v = inkling_attn_prologue_decode(qkvr, ..., mxfp8_quant=True)
# after
aligned = dq % 128 == 0 and dkv % 128 == 0
q, k, v = inkling_attn_prologue_decode(qkvr, ..., mxfp8_quant=mxfp8_quant and aligned)

Defensive patterns

Strategy: validation

Validate before calling

mxfp8_quant = mxfp8_quant and dq % 128 == 0 and dkv % 128 == 0

Prevention

Apply the same alignment gate to prefill and decode paths so they fail (or degrade) consistently
Log once when falling back from MXFP8 due to alignment

When it happens

Trigger: Calling inkling_attn_prologue_decode with mxfp8_quant=True on a model with head dims not divisible by 128 — the first decode step after prefill, i.e. the error appears one step later than the extend-path variant (372/370).

Common situations: MXFP8 enabled for models with head_dim 64; configurations where prefill ran non-quantized but decode routed into the quantized prologue.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3ea89fbbc0d7138e. Report an issue: GitHub.