sgl-project/sglang · error · ValueError

The quantization block size of weight must have 2 dimensions

Error message

The quantization block size of weight must have 2 dimensions, but got {len(weight_block_size)} dimensions.

What it means

Fp8LinearConfig requires weight_block_size to be a 2-element iterable [block_n, block_k] because FP8 block quantization scales form a 2D grid over the weight matrix. Passing a scalar, a 1-element or 3+-element list fails this validation.

Source

Thrown at python/sglang/srt/layers/quantization/fp8.py:268

        self.ignored_layers = ignored_layers or []
        if ignored_layers_str := envs.SGLANG_FP8_IGNORED_LAYERS.get():
            self.ignored_layers.extend(
                [
                    layer.strip()
                    for layer in ignored_layers_str.split(",")
                    if layer.strip()
                ]
            )
        self.packed_modules_mapping = packed_modules_mapping or {}
        self.use_mxfp8 = use_mxfp8
        self.kv_cache_quant_algo = kv_cache_quant_algo
        if weight_block_size is not None:
            if not is_checkpoint_fp8_serialized:
                raise ValueError(
                    "The block-wise quantization only supports fp8-serialized checkpoint for now."
                )
            if len(weight_block_size) != 2:
                raise ValueError(
                    f"The quantization block size of weight must have 2 dimensions, but got {len(weight_block_size)} dimensions."
                )
            if activation_scheme != "dynamic":
                raise ValueError(
                    f"The block-wise quantization only supports dynamic activation scheme for now, but got {activation_scheme} activation scheme."
                )
        if self.use_mxfp8:
            if weight_block_size is None:
                weight_block_size = [1, 32]
            elif weight_block_size != [1, 32]:
                raise ValueError("MXFP8 requires weight_block_size=[1, 32].")
        self.weight_block_size = weight_block_size

    def get_name(self) -> str:
        return "mxfp8" if self.use_mxfp8 else "fp8"

    @classmethod
    def get_supported_act_dtypes(cls) -> List[torch.dtype]:

View on GitHub (pinned to 0132848349)

Solutions

1. Set weight_block_size to the checkpoint's actual 2D block, e.g. [128, 128]
2. Verify in the model's config.json and fix the quantization_config entry
3. If loading a real DeepSeek FP8 checkpoint, its shipped config already has the correct [128,128] — prefer it over modified copies

Example fix

// before
"weight_block_size": 128
// after
"weight_block_size": [128, 128]

wbs = qcfg.get("weight_block_size")
if wbs is not None and (not isinstance(wbs, (list, tuple)) or len(wbs) != 2):
    raise SystemExit(f"weight_block_size must be [block_n, block_k], got {wbs!r}")

def is_two_element_block_size(w) -> bool:
    return isinstance(w, (list, tuple)) and len(w) == 2 and all(isinstance(x, int) and x > 0 for x in w)

When it happens

Trigger: Fp8LinearConfig(weight_block_size=128) or weight_block_size=[128] / [128,128,128]; typically from a config.json that stored a scalar block size or an int-parsed value.

Common situations: Configs written by external quantization tools that use a single block size parameter; hand-edited quantization_config; schema drift between checkpoint producers.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

• The hpc_ops MoE runner backend only supports FP8-quantized M
• Unsupported activation scheme {activation_scheme}
• Weight input_size_per_partition = {input_size_per_partition}
• Weight output_partition_size = {output_partition_size} is no
• The output_size of gate's and up's weight = {intermediate_si

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/c1bd2bb772f9e0e4. Report an issue: GitHub.