sgl-project/sglang · error · ValueError

MXFP8 fused prologue requires head_dim-aligned Q/K/V.

Error message

MXFP8 fused prologue requires head_dim-aligned Q/K/V.

What it means

inkling_attn_prologue_verify is the verification (target-verify) variant of the fused attention prologue; when mxfp8_quant=True it quantizes Q/K/V to MXFP8, which requires the 128-byte-blocked microscaling format — hence dq and dkv must both be multiples of 128. The check dq % 128 != 0 or dkv % 128 != 0 rejects non-aligned head dims before allocating float8_e4m3 buffers, because scale factors are computed per 32-element block within 128-element superblocks.

Source

Thrown at python/sglang/kernels/ops/attention/inkling_attn_prologue.py:92

    dq: int,
    dkv: int,
    draft_token_num: int,
    activation: str | None = None,
    use_residual: bool = True,
    do_store: bool = True,
    mxfp8_quant: bool = False,
    sfk: torch.Tensor | None = None,
    sfv: torch.Tensor | None = None,
    page_size: int = 128,
    log_scaling_tau: torch.Tensor | None = None,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:
    """Returns fresh contiguous (q_normed, k_normed, v_conv) [T, dq/dkv];
    KV rows are also scattered into k_buf/v_buf at ``loc`` (the attention call
    should pass save_kv_cache=False)."""
    t = qkvr.shape[0]
    if mxfp8_quant:
        if dq % 128 != 0 or dkv % 128 != 0:
            raise ValueError("MXFP8 fused prologue requires head_dim-aligned Q/K/V.")
        if sfk is None or sfv is None:
            raise ValueError("MXFP8 fused prologue requires K/V scale buffers.")
        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)
        if sfk.shape != sf_shape or sfv.shape != sf_shape:
            raise ValueError(
                "MXFP8 fused prologue requires interleaved K/V scale buffers "
                f"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}."
            )
        if not sfk.is_contiguous() or not sfv.is_contiguous():
            raise ValueError(
                "MXFP8 fused prologue requires contiguous interleaved SFK/SFV."
            )
        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)
        sfq_u8 = torch.empty(
            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device
        )
        sfk_u8 = sfk.view(torch.uint8)
        sfv_u8 = sfv.view(torch.uint8)

View on GitHub (pinned to 0132848349)

Solutions

1. Disable mxfp8_quant (fall back to the non-quantized prologue) for this layer/model
2. Use a model configuration with head dims that are multiples of 128 (e.g. 128 or 256)
3. Gate the MXFP8 path on (dq % 128 == 0 and dkv % 128 == 0) in the caller so it degrades gracefully

Example fix

# before
q, k, v = inkling_attn_prologue_verify(qkvr, ..., mxfp8_quant=True)
# after
use_mxfp8 = (dq % 128 == 0 and dkv % 128 == 0)
q, k, v = inkling_attn_prologue_verify(qkvr, ..., mxfp8_quant=use_mxfp8)

mxfp8_quant = mxfp8_quant and dq % 128 == 0 and dkv % 128 == 0

When it happens

Trigger: Calling inkling_attn_prologue_verify with mxfp8_quant=True on a model whose head dims (dq, dkv) are not multiples of 128 — e.g. dq=64, dkv=128, or dq=96 — such as smaller Inkling configurations or head_dim-64 attention layers routed into the fused MXFP8 path.

Common situations: Enabling MXFP8 quantization for a model with head_dim 64/80/96; a config knob like quantization_kv_cache=mxfp8 applied globally across layers with heterogeneous head dims; new model checkpoints with non-128-aligned dims tested against the fused prologue.

Related errors

• MXFP8 fused prologue requires K/V scale buffers.
• MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.
• MXFP8 KV cache requires head_dim divisible by {self.MXFP8_SC
• MXFP8 fused prologue requires interleaved K/V scale buffers
• MXFP8 fused prologue requires contiguous interleaved SFK/SFV

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/09a467403ccb6a54. Report an issue: GitHub.