sgl-project/sglang · error · ValueError

Comfy NVFP4 layer {prefix!r} has incompatible weight/scale s

Error message

Comfy NVFP4 layer {prefix!r} has incompatible weight/scale shapes: {weight_shape} and {scale_shape}

What it means

For nvfp4 layers, after unpacking (logical_input_size = weight_shape[1]*2) the checker requires logical_input_size divisible by 16 (the FP4 block size) and weight_scale shape exactly (weight_shape[0], logical_input_size//16) — one FP8 scale per 16 quantized values. Any deviation means scale and weight layouts disagree and dequantization would be incorrect.

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/quantization_utils.py:272

            if weight_dtype != "U8" or scale_dtype != "F8_E4M3":
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs U8 packed weights and "
                    f"FP8 block scales, got {weight_dtype} and {scale_dtype}"
                )
            if scale_2_dtype != "F32" or scale_2_shape not in ((), (1,)):
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs a scalar F32 "
                    f"weight_scale_2, got {scale_2_dtype}{scale_2_shape}"
                )
            if len(weight_shape) != 2:
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, "
                    f"got {weight_shape}"
                )
            logical_input_size = weight_shape[1] * 2
            expected_scale_shape = (weight_shape[0], logical_input_size // 16)
            if logical_input_size % 16 or scale_shape != expected_scale_shape:
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} has incompatible weight/scale "
                    f"shapes: {weight_shape} and {scale_shape}"
                )
            pre_quant_scale_key = f"{prefix}.pre_quant_scale"
            marker["_has_pre_quant_scale"] = pre_quant_scale_key in checkpoint_meta
            if marker["_has_pre_quant_scale"]:
                pre_scale_dtype, pre_scale_shape = checkpoint_meta[pre_quant_scale_key]
                if pre_scale_dtype not in ("BF16", "F16", "F32") or (
                    pre_scale_shape != (logical_input_size,)
                ):
                    raise ValueError(
                        f"Comfy NVFP4 layer {prefix!r} has an incompatible "
                        f"pre_quant_scale: {pre_scale_dtype}{pre_scale_shape}"
                    )
            continue
        if marker_format != "int8_tensorwise":
            continue
        weight_dtype, weight_shape = checkpoint_meta[f"{prefix}.weight"]

View on GitHub (pinned to 0132848349)

Solutions

1. Print weight_shape and scale_shape for the failing prefix and compute expected (shape[0], shape[1]*2//16)
2. Re-export with block size 16 scales matching the packed weight layout
3. If the weight is transposed relative to the scale, fix the export orientation
4. If in_features isn't a multiple of 16, exclude that layer from nvfp4 quantization

Example fix

# before: weight (3072, 2048), scale (3072, 2048)  # block size wrong
# after: scale (3072, 4096//16) = (3072, 256)

_, w_shape = checkpoint_meta[f"{prefix}.weight"]
_, s_shape = checkpoint_meta[f"{prefix}.weight_scale"]
logical_in = w_shape[1] * 2
expected = (w_shape[0], logical_in // 16)
assert logical_in % 16 == 0 and s_shape == expected, f"expected scale {expected}, got {s_shape}"

def nvfp4_scale_shape_matches(w_shape, s_shape) -> bool:
    logical_in = w_shape[1] * 2
    return logical_in % 16 == 0 and s_shape == (w_shape[0], logical_in // 16)

When it happens

Trigger: `inspect_comfy_quant_markers` on an nvfp4 layer where the scale tensor's second dim doesn't equal in_features/16, the first dim doesn't equal out_features, or in_features isn't a multiple of 16. E.g. scale saved with block size 32, or a transposed weight.

Common situations: Exporter using a different block size than 16; transposed weight/scale conventions between exporter versions; checkpoint resized or pruned after quantization; partial-tensor merging.

Related errors

• Comfy NVFP4 layer {prefix!r} has an incompatible pre_quant_s
• Shape mismatch: A K={k}, B K={b.shape[1] * 2}
• Comfy W4A4 layer {prefix!r} has input size {logical_input_si
• Comfy NVFP4 layer {prefix!r} needs U8 packed weights and FP8
• Comfy NVFP4 layer {prefix!r} needs a scalar F32 weight_scale

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e502ab400cf92305. Report an issue: GitHub.