sgl-project/sglang · error · ValueError

MXFP8 fused decode prologue requires contiguous interleaved

Error message

MXFP8 fused decode prologue requires contiguous interleaved SFK/SFV.

What it means

In the decode path, sfk and sfv must be contiguous tensors because the fused kernel writes the interleaved scale layout with dense-stride indexing. The check not sfk.is_contiguous() or not sfv.is_contiguous() runs after shape validation and before the fp8 output allocation; strided views would cause scales to be written at wrong memory offsets, corrupting the MXFP8 cache.

Source

Thrown at python/sglang/kernels/ops/attention/inkling_attn_prologue.py:335

    The k/v conv caches are shift-updated in place (fused_decode_update
    semantics). With ``do_store`` the KV rows are scattered into k_buf/v_buf at
    ``loc``; MXFP8 mode also quantizes Q and writes interleaved K/V scales."""
    t = qkvr.shape[0]
    if mxfp8_quant:
        if dq % 128 != 0 or dkv % 128 != 0:
            raise ValueError(
                "MXFP8 fused decode prologue requires head_dim-aligned Q/K/V."
            )
        if sfk is None or sfv is None:
            raise ValueError("MXFP8 fused decode prologue requires K/V scale buffers.")
        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)
        if sfk.shape != sf_shape or sfv.shape != sf_shape:
            raise ValueError(
                "MXFP8 fused decode prologue requires interleaved K/V scale buffers "
                f"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}."
            )
        if not sfk.is_contiguous() or not sfv.is_contiguous():
            raise ValueError(
                "MXFP8 fused decode prologue requires contiguous interleaved SFK/SFV."
            )
        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)
        sfq_u8 = torch.empty(
            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device
        )
        sfk_u8 = sfk.view(torch.uint8)
        sfv_u8 = sfv.view(torch.uint8)
    else:
        q_out = torch.empty(t, dq, dtype=qkvr.dtype, device=qkvr.device)
        sfq_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)
        sfk_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)
        sfv_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)
    k_out = torch.empty(t, dkv, dtype=qkvr.dtype, device=qkvr.device)
    v_out = torch.empty(t, dkv, dtype=qkvr.dtype, device=qkvr.device)
    if activation == "swish":
        activation = "silu"
    use_silu = activation in ("silu", "swish")

View on GitHub (pinned to 0132848349)

Solutions

1. Call .contiguous() on sfk/sfv before the decode call
2. Store per-layer scale buffers as dense standalone tensors
3. Add an is_contiguous() assert in the pool accessor

Example fix

# before
sfk, sfv = layer_view_of_pool  # non-contiguous
# after
sfk, sfv = layer_view_of_pool.contiguous()

if mxfp8_quant:\n    sfk = sfk.contiguous(); sfv = sfv.contiguous()

When it happens

Trigger: Passing strided or sliced sfk/sfv views into inkling_attn_prologue_decode — e.g. buffers carved out of a stacked multi-layer scale pool with narrow/slice, or after a transpose-based layout fix.

Common situations: View-based per-layer scale buffers over one big allocation; buffers returned from a cache that stores transposed layouts internally.

Related errors

• MXFP8 fused prologue requires contiguous interleaved SFK/SFV
• MXFP8 fused decode prologue requires K/V scale buffers.
• MXFP8 fused decode prologue requires interleaved K/V scale b
• MXFP8 fused prologue requires K/V scale buffers.
• MXFP8 fused prologue requires interleaved K/V scale buffers

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b4f3afba47314526. Report an issue: GitHub.