sgl-project/sglang · error · ValueError

MXFP8 fused prologue requires contiguous interleaved SFK/SFV

Error message

MXFP8 fused prologue requires contiguous interleaved SFK/SFV.

What it means

The MXFP8 scale buffers sfk/sfv must be contiguous because the fused prologue kernel indexes them with the interleaved layout assuming dense row-major memory; a non-contiguous view (slice/permute of a larger buffer) would make the written scales garbage. The check not sfk.is_contiguous() or not sfv.is_contiguous() runs after the shape check, right before allocating the fp8 output tensors.

Source

Thrown at python/sglang/kernels/ops/attention/inkling_attn_prologue.py:102

    log_scaling_tau: torch.Tensor | None = None,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor | None]:
    """Returns fresh contiguous (q_normed, k_normed, v_conv) [T, dq/dkv];
    KV rows are also scattered into k_buf/v_buf at ``loc`` (the attention call
    should pass save_kv_cache=False)."""
    t = qkvr.shape[0]
    if mxfp8_quant:
        if dq % 128 != 0 or dkv % 128 != 0:
            raise ValueError("MXFP8 fused prologue requires head_dim-aligned Q/K/V.")
        if sfk is None or sfv is None:
            raise ValueError("MXFP8 fused prologue requires K/V scale buffers.")
        sf_shape = (k_buf.shape[0] // page_size, dkv // 128, 32, page_size // 32, 4)
        if sfk.shape != sf_shape or sfv.shape != sf_shape:
            raise ValueError(
                "MXFP8 fused prologue requires interleaved K/V scale buffers "
                f"with shape {sf_shape}, got {tuple(sfk.shape)} and {tuple(sfv.shape)}."
            )
        if not sfk.is_contiguous() or not sfv.is_contiguous():
            raise ValueError(
                "MXFP8 fused prologue requires contiguous interleaved SFK/SFV."
            )
        q_out = torch.empty(t, dq, dtype=torch.float8_e4m3fn, device=qkvr.device)
        sfq_u8 = torch.empty(
            (t, dq // 128, 128 // 32), dtype=torch.uint8, device=qkvr.device
        )
        sfk_u8 = sfk.view(torch.uint8)
        sfv_u8 = sfv.view(torch.uint8)
    else:
        q_out = torch.empty(t, dq, dtype=qkvr.dtype, device=qkvr.device)
        sfq_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)
        sfk_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)
        sfv_u8 = torch.empty(0, dtype=torch.uint8, device=qkvr.device)
    k_out = torch.empty(t, dkv, dtype=qkvr.dtype, device=qkvr.device)
    v_out = torch.empty(t, dkv, dtype=qkvr.dtype, device=qkvr.device)
    if activation == "swish":
        activation = "silu"
    use_silu = activation in ("silu", "swish")

View on GitHub (pinned to 0132848349)

Solutions

1. Call .contiguous() on sfk/sfv before passing them (or better, fix the allocation so each buffer is dense)
2. Allocate per-layer scale buffers instead of slicing a concatenated pool tensor
3. Check sfk.is_contiguous() in your pool getter and densify once at setup, not per step

Example fix

# before
sfk, sfv = pool.scales[:, layer_idx]  # non-contiguous views
# after
sfk, sfv = pool.scales[layer_idx].contiguous()

if mxfp8_quant:\n    sfk = sfk.contiguous(); sfv = sfv.contiguous()

def contiguous_scales(t: torch.Tensor) -> torch.Tensor:\n    return t if t.is_contiguous() else t.contiguous()

When it happens

Trigger: Passing sfk/sfv that are strided views — e.g. sfk = big_buffer[:, 1] or a permuted/reshaped-with-stride tensor — into inkling_attn_prologue_verify with mxfp8_quant=True.

Common situations: Slicing one layer's scale buffer out of a fused multi-layer pool tensor; using .view() where it produces non-contiguous strides; narrow() on the page dimension.

Related errors

• MXFP8 fused prologue requires interleaved K/V scale buffers
• MXFP8 fused decode prologue requires K/V scale buffers.
• MXFP8 fused decode prologue requires contiguous interleaved
• MXFP8 fused prologue requires K/V scale buffers.
• MXFP8 fused decode prologue requires interleaved K/V scale b

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/995213a7f3b16620. Report an issue: GitHub.