sgl-project/sglang · error · RuntimeError

--fp8-gemm-backend=deep_gemm cannot serve MXFP8 weight shape

Error message

--fp8-gemm-backend=deep_gemm cannot serve MXFP8 weight shape ({n}, {k}) (needs N % 64 == 0 and K % 128 == 0), and this device has no FlashInfer MXFP8 fallback kernel.

What it means

DeepGEMM's MXFP8 kernels require weight shapes with N % 64 == 0 and K % 128 == 0. When the shape violates this and the GPU is not Blackwell-with-FlashInfer (which provides a fallback via block_scale_interleave), SGLang raises instead of silently running a broken GEMM.

Source

Thrown at python/sglang/srt/layers/quantization/fp8.py:797

            scale_u8 = layer.weight_scale_inv.data
            # block_scale_interleave may pad and/or reshape scales,
            # so store swizzled scales separately to keep weight update working
            copy_or_rebind_param(
                layer,
                "weight_scale_inv_swizzled",
                block_scale_interleave(scale_u8.contiguous()).contiguous(),
            )
        elif backend.is_deep_gemm():
            from sglang.srt.layers.deep_gemm_wrapper.configurer import (
                DEEPGEMM_SCALE_UE8M0,
            )

            n, k = layer.weight.shape
            scale_u8 = layer.weight_scale_inv.data
            layer.weight_scale_inv_swizzled = None
            if n % 64 != 0 or k % 128 != 0:
                if not (is_blackwell_supported() and is_flashinfer_available()):
                    raise RuntimeError(
                        f"--fp8-gemm-backend=deep_gemm cannot serve MXFP8 weight shape "
                        f"({n}, {k}) (needs N % 64 == 0 and K % 128 == 0), and this "
                        "device has no FlashInfer MXFP8 fallback kernel."
                    )
                from flashinfer import block_scale_interleave

                copy_or_rebind_param(
                    layer,
                    "weight_scale_inv_swizzled",
                    block_scale_interleave(scale_u8.contiguous()).contiguous(),
                )
            scale_fp32 = (
                (scale_u8.contiguous().view(-1).to(torch.int32) << 23)
                .view(torch.float32)
                .view(n, k // 32)
            )
            if DEEPGEMM_SCALE_UE8M0:
                # Pre-packed; GEMM must be called with disable_ue8m0_cast=True.

View on GitHub (pinned to 0132848349)

Solutions

1. Remove --fp8-gemm-backend=deep_gemm (or set auto) so a compatible MXFP8 backend is chosen
2. Install FlashInfer and run on a Blackwell (SM100) GPU to enable the swizzle fallback path
3. Use a checkpoint variant whose weight shapes satisfy N%64==0 and K%128==0

Example fix

# before
python -m sglang.launch_server --model ... --fp8-gemm-backend=deep_gemm
# after
python -m sglang.launch_server --model ...  # default auto backend selection

n, k = layer_weight_shape
if args.fp8_gemm_backend == "deep_gemm" and (n % 64 or k % 128):
    assert not (n % 64 or k % 128), "deep_gemm mxfp8 needs N%64==0 and K%128==0; drop the flag or use Blackwell+FlashInfer"

def deep_gemm_mxfp8_shape_ok(shape) -> bool:
    n, k = shape
    return n % 64 == 0 and k % 128 == 0

When it happens

Trigger: --fp8-gemm-backend=deep_gemm with an MXFP8 linear layer whose weight has N not multiple of 64 or K not multiple of 128, on a non-Blackwell GPU or without FlashInfer installed; triggered during _process_mxfp8_linear_weight_scale at weight load.

Common situations: Running MXFP8-quantized models (e.g. Llama MX checkpoints) with deep_gemm forced on pre-Blackwell hardware (H100) or in containers lacking flashinfer; small/odd vocab or intermediate dims.

Related errors

• MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer
• MXFP8 dense GEMM requested via --fp8-gemm-backend=flashinfer
• MXFP8 MoE quantization requires SM100 or ROCm gfx95 (gfx942
• MXFP8 fused prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused prologue requires K/V scale buffers.

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/faf1e961d8246da2. Report an issue: GitHub.