sgl-project/sglang · error · ValueError

The block-wise quantization only supports fp8-serialized che

Error message

The block-wise quantization only supports fp8-serialized checkpoint for now.

What it means

Block-wise FP8 quantization (weight_block_size set, as in DeepSeek-V2/V3 style checkpoints) requires the checkpoint to actually be FP8-serialized, because block scales can only be consumed from an FP8 checkpoint layout. Fp8LinearConfig raises when weight_block_size is provided together with is_checkpoint_fp8_serialized=False.

Source

Thrown at python/sglang/srt/layers/quantization/fp8.py:264

            log_info_on_rank0(logger, "Detected fp8 checkpoint.")
        if activation_scheme not in ACTIVATION_SCHEMES:
            raise ValueError(f"Unsupported activation scheme {activation_scheme}")
        self.activation_scheme = activation_scheme
        self.ignored_layers = ignored_layers or []
        if ignored_layers_str := envs.SGLANG_FP8_IGNORED_LAYERS.get():
            self.ignored_layers.extend(
                [
                    layer.strip()
                    for layer in ignored_layers_str.split(",")
                    if layer.strip()
                ]
            )
        self.packed_modules_mapping = packed_modules_mapping or {}
        self.use_mxfp8 = use_mxfp8
        self.kv_cache_quant_algo = kv_cache_quant_algo
        if weight_block_size is not None:
            if not is_checkpoint_fp8_serialized:
                raise ValueError(
                    "The block-wise quantization only supports fp8-serialized checkpoint for now."
                )
            if len(weight_block_size) != 2:
                raise ValueError(
                    f"The quantization block size of weight must have 2 dimensions, but got {len(weight_block_size)} dimensions."
                )
            if activation_scheme != "dynamic":
                raise ValueError(
                    f"The block-wise quantization only supports dynamic activation scheme for now, but got {activation_scheme} activation scheme."
                )
        if self.use_mxfp8:
            if weight_block_size is None:
                weight_block_size = [1, 32]
            elif weight_block_size != [1, 32]:
                raise ValueError("MXFP8 requires weight_block_size=[1, 32].")
        self.weight_block_size = weight_block_size

    def get_name(self) -> str:

View on GitHub (pinned to 0132848349)

Solutions

1. Use a genuinely FP8-serialized checkpoint (e.g. deepseek-ai DeepSeek-V3 FP8 releases) that includes weight scales and is_checkpoint_fp8_serialized=True
2. If quantizing on the fly from BF16, remove weight_block_size (or the quantization_config block) so per-tensor/on-the-fly paths are used instead
3. Regenerate the quantization_config with quant_method="fp8", activation_scheme="dynamic", and FP8 weights

Example fix

// before
{"quant_method": "fp8", "weight_block_size": [128,128]}  // BF16 weights
// after
{"quant_method": "fp8", "activation_scheme": "dynamic", "weight_block_size": [128,128], "fmt": "e4m3"}  // true FP8 checkpoint
// or: remove quantization_config entirely for BF16

qcfg = model_config.quantization_config
if qcfg.get("weight_block_size") and not qcfg.get("is_checkpoint_fp8_serialized", True):
    raise SystemExit("block-wise FP8 requires an FP8-serialized checkpoint; use a real FP8 release")

def block_quant_config_is_consistent(qcfg: dict) -> bool:
    if qcfg.get("weight_block_size") is None:
        return True
    return bool(qcfg.get("is_checkpoint_fp8_serialized")) or qcfg.get("fmt") == "e4m3"

When it happens

Trigger: Fp8LinearConfig(weight_block_size=[128,128], is_checkpoint_fp8_serialized=False) — e.g. a config where quant_method is fp8 with block sizes but the weight files are BF16/FP16, or a quantization_config that omitted "fmt": "e4m3" so serialization detection failed.

Common situations: On-the-fly FP8 quantization of a BF16 DeepSeek checkpoint that carries leftover weight_block_size in config.json; manually merged configs where is_checkpoint_fp8_serialized was defaulted to False.

Related errors

• Unsupported activation scheme {activation_scheme}
• The block-wise quantization only supports dynamic activation
• Found static activation scheme for checkpoint that was not s
• kv_scales supplied but unified_kv is {unified_kv.dtype}, exp
• int32-packed scale buffers require scale_ue8m0=True

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fbbc7211145a659b. Report an issue: GitHub.