sgl-project/sglang · error · ValueError

MXFP8 requires weight_block_size=[1, 32].

Error message

MXFP8 requires weight_block_size=[1, 32].

What it means

MXFP8 (microscaling FP8, OCP MX format) uses a fixed 1x32 block (32 elements per scale), so any weight_block_size other than [1, 32] is rejected by Fp8LinearConfig when use_mxfp8 is true. [1, 32] is the only legal MX block.

Source

Thrown at python/sglang/srt/layers/quantization/fp8.py:279

        self.kv_cache_quant_algo = kv_cache_quant_algo
        if weight_block_size is not None:
            if not is_checkpoint_fp8_serialized:
                raise ValueError(
                    "The block-wise quantization only supports fp8-serialized checkpoint for now."
                )
            if len(weight_block_size) != 2:
                raise ValueError(
                    f"The quantization block size of weight must have 2 dimensions, but got {len(weight_block_size)} dimensions."
                )
            if activation_scheme != "dynamic":
                raise ValueError(
                    f"The block-wise quantization only supports dynamic activation scheme for now, but got {activation_scheme} activation scheme."
                )
        if self.use_mxfp8:
            if weight_block_size is None:
                weight_block_size = [1, 32]
            elif weight_block_size != [1, 32]:
                raise ValueError("MXFP8 requires weight_block_size=[1, 32].")
        self.weight_block_size = weight_block_size

    def get_name(self) -> str:
        return "mxfp8" if self.use_mxfp8 else "fp8"

    @classmethod
    def get_supported_act_dtypes(cls) -> List[torch.dtype]:
        return [torch.bfloat16, torch.half]

    def get_min_capability(self) -> int:
        if is_npu():
            return 0  # NPU bypasses CUDA capability checks
        if _is_musa:
            return 31
        if self.use_mxfp8 and _is_hip and _is_gfx95_supported:
            return 95
        if self.use_mxfp8 and _mxfp8_to_block_fp8_required:
            return 94

View on GitHub (pinned to 0132848349)

Solutions

1. Remove weight_block_size from the quantization_config (it defaults to [1,32] for MXFP8) or set it exactly to [1, 32]
2. Use a genuine MXFP8 checkpoint whose config already omits or sets [1,32]

Example fix

// before
{"quant_method":"mxfp8","weight_block_size":[128,128]}
// after
{"quant_method":"mxfp8"}  // weight_block_size defaults to [1,32]

if qcfg.get("use_mxfp8") or qcfg.get("quant_method") == "mxfp8":
    wbs = qcfg.get("weight_block_size")
    if wbs is not None and list(wbs) != [1, 32]:
        qcfg.pop("weight_block_size")  # default [1,32]

def is_legal_mxfp8_block(wbs) -> bool:
    return wbs is None or list(wbs) == [1, 32]

When it happens

Trigger: Fp8LinearConfig(use_mxfp8=True, weight_block_size=[128,128]) — a config that enables mxfp8 (often auto-detected from quant_method "mxfp8" or compressed-tensors MX schemes) but also carries a legacy FP8 block size.

Common situations: Editing a DeepSeek FP8 [128,128] config to switch to MXFP8 without removing the block size; checkpoints with both keys present; mixing config fragments from different quant formats.

Related errors

• MXFP8 fused prologue requires head_dim-aligned Q/K/V.
• MXFP8 fused prologue requires K/V scale buffers.
• MXFP8 fused decode prologue requires head_dim-aligned Q/K/V.
• kitchen_int8 group_size must be one of {_SUPPORTED_GROUP_SIZ
• Unsupported Comfy INT8 format for {prefix!r}: {marker.get('f

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/bb686cfe19ce8af5. Report an issue: GitHub.