sgl-project/sglang · error · ValueError

Comfy NVFP4 layer {prefix!r} needs U8 packed weights and FP8

Error message

Comfy NVFP4 layer {prefix!r} needs U8 packed weights and FP8 block scales, got {weight_dtype} and {scale_dtype}

What it means

For `nvfp4` format layers, the checker requires the packed weight tensor dtype to be U8 (two 4-bit values per byte) and the block-scale tensor dtype to be F8_E4M3 (FP8). This matches the NVFP4 kernel contract: UINT4x2 weights with FP8 E4M3 16-element block scales plus a global F32 scale. Any other dtype pair means the checkpoint wasn't produced by a compliant NVFP4 quantizer.

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/quantization_utils.py:255

            convrot_group_size = int(marker.get("convrot_groupsize", 256))
            if convrot_group_size not in (16, 64, 256):
                raise ValueError(
                    f"Comfy W4A4 layer {prefix!r} has unsupported "
                    f"convrot_groupsize={convrot_group_size}"
                )
            if logical_input_size % 64 or logical_input_size % convrot_group_size:
                raise ValueError(
                    f"Comfy W4A4 layer {prefix!r} has input size "
                    f"{logical_input_size}, incompatible with quant_group_size=64 "
                    f"and convrot_groupsize={convrot_group_size}"
                )
            continue
        if marker_format == "nvfp4":
            weight_dtype, weight_shape = checkpoint_meta[f"{prefix}.weight"]
            scale_dtype, scale_shape = checkpoint_meta[f"{prefix}.weight_scale"]
            scale_2_dtype, scale_2_shape = checkpoint_meta[f"{prefix}.weight_scale_2"]
            if weight_dtype != "U8" or scale_dtype != "F8_E4M3":
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs U8 packed weights and "
                    f"FP8 block scales, got {weight_dtype} and {scale_dtype}"
                )
            if scale_2_dtype != "F32" or scale_2_shape not in ((), (1,)):
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs a scalar F32 "
                    f"weight_scale_2, got {scale_2_dtype}{scale_2_shape}"
                )
            if len(weight_shape) != 2:
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, "
                    f"got {weight_shape}"
                )
            logical_input_size = weight_shape[1] * 2
            expected_scale_shape = (weight_shape[0], logical_input_size // 16)
            if logical_input_size % 16 or scale_shape != expected_scale_shape:
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} has incompatible weight/scale "

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect the safetensors header for the failing prefix and confirm actual dtypes of weight and weight_scale
2. Re-export the model with a quantizer that packs NVFP4 weights as U8 and block scales as F8_E4M3 (e.g. the matching Comfy/tensorrt export recipe)
3. If the format marker is wrong (the layer was quantized as another format), fix the marker's 'format' field
4. Upgrade SGLang if a newer build accepts the dtype combination you have

Example fix

# before: weight dtype I8, scale dtype BF16
# after (re-export): weight dtype U8 (packed fp4x2), weight_scale dtype F8_E4M3

wd, _ = checkpoint_meta[f"{prefix}.weight"]
sd, _ = checkpoint_meta[f"{prefix}.weight_scale"]
if marker.get("format") == "nvfp4":
    assert wd == "U8" and sd == "F8_E4M3", f"{prefix}: got {wd}/{sd}"

def is_valid_nvfp4_dtypes(meta: dict, prefix: str) -> bool:
    return (
        meta[f"{prefix}.weight"][0] == "U8"
        and meta[f"{prefix}.weight_scale"][0] == "F8_E4M3"
    )

When it happens

Trigger: `inspect_comfy_quant_markers` on a checkpoint where a marker has format 'nvfp4' but `{prefix}.weight` is not dtype U8 or `{prefix}.weight_scale` is not F8_E4M3 — e.g. weights saved unpacked as I8/INT4 tensors, or scales stored as F16/BF16/F32.

Common situations: Mixing checkpoints quantized with different NVFP4 export recipes; an older/newer exporter that saved FP8 scales as BF16; converting a checkpoint through a tool that re-casts dtypes; mislabeling a layer's format marker as nvfp4 when it is really w4a8 or int8.

Related errors

• Comfy NVFP4 layer {prefix!r} needs a scalar F32 weight_scale
• Type must match: {self.a_dtype} != {self.b_dtype}
• int32-packed scale buffers require scale_ue8m0=True
• scale_ue8m0=True requires an int32-packed output_s
• Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, got {

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/85a8240a7e88a92e. Report an issue: GitHub.