sgl-project/sglang · error · ValueError

Comfy W4A8 layer {prefix!r} needs I8 weights and FP8 group s

Error message

Comfy W4A8 layer {prefix!r} needs I8 weights and FP8 group scales, got {weight_dtype} and {scale_dtype}

What it means

A Comfy W4A8 layer must store its packed weights as int8 (dtype string 'I8' in safetensors metadata) and its per-group scales as FP8 e4m3 ('F8_E4M3'). This error means one of those tensors has a different dtype, indicating the checkpoint was quantized with an incompatible scheme.

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/quantization_utils.py:177

                f"Comfy layer {prefix!r} is missing checkpoint tensors: "
                f"{sorted(missing)}"
            )
        if marker_format == "float8_e4m3fn":
            marker["_activation_scheme"] = (
                "static" if f"{prefix}.input_scale" in checkpoint_meta else "dynamic"
            )
            continue
        if marker_format == "asym_w4a8_int8":
            weight_dtype, weight_shape = checkpoint_meta[f"{prefix}.weight"]
            scale_dtype, scale_shape = checkpoint_meta[f"{prefix}.weight_s_rel"]
            channel_dtype, channel_shape = checkpoint_meta[f"{prefix}.weight_s_channel"]
            group_size = int(marker.get("group_size", 16))
            if group_size < 4:
                raise ValueError(
                    f"Comfy W4A8 layer {prefix!r} has invalid group_size={group_size}"
                )
            if weight_dtype != "I8" or scale_dtype != "F8_E4M3":
                raise ValueError(
                    f"Comfy W4A8 layer {prefix!r} needs I8 weights and FP8 "
                    f"group scales, got {weight_dtype} and {scale_dtype}"
                )
            if channel_dtype != "F32":
                raise ValueError(
                    f"Comfy W4A8 layer {prefix!r} needs F32 channel scales, "
                    f"got {channel_dtype}"
                )
            if len(weight_shape) != 2:
                raise ValueError(
                    f"Comfy W4A8 layer {prefix!r} needs a 2D packed weight, "
                    f"got {weight_shape}"
                )
            logical_input_size = weight_shape[1] * 2
            expected_scale_shape = (weight_shape[0], logical_input_size // group_size)
            if scale_shape != expected_scale_shape or channel_shape != (
                weight_shape[0],
            ):

View on GitHub (pinned to 0132848349)

Solutions

1. Verify the checkpoint was produced by the Comfy W4A8 (asym_w4a8_int8) exporter, not another quantizer
2. Check dtypes: safetensors metadata for {prefix}.weight should read I8 and {prefix}.weight_s_rel should read F8_E4M3
3. Re-quantize/re-export the model with the correct W4A8 recipe, or load it with the matching quant backend
4. If the model is genuinely a different format, update the marker format string so it routes to the right handler

Example fix

# before: weight dtype BF16, scale dtype F32  -> raises
# after: weight dtype I8 (packed int8), scale dtype F8_E4M3
quantize_comfy_w4a8(model, out_dir="out")  # exporter writes correct dtypes

wdt, _ = meta[f"{prefix}.weight"]; sdt, _ = meta[f"{prefix}.weight_s_rel"]
if wdt != "I8" or sdt != "F8_E4M3":
    raise ValueError(f"not a Comfy W4A8 layer: {wdt}/{sdt}")

def is_comfy_w4a8_layer(meta: dict, prefix: str) -> bool:
    w = meta.get(f"{prefix}.weight"); s = meta.get(f"{prefix}.weight_s_rel")
    return w is not None and s is not None and w[0] == "I8" and s[0] == "F8_E4M3"

When it happens

Trigger: inspect_comfy_quant_markers reads {prefix}.weight and {prefix}.weight_s_rel metadata for an asym_w4a8_int8 layer and finds weight_dtype != 'I8' or scale_dtype != 'F8_E4M3' (e.g. F16/BF16 weights or F32 scales).

Common situations: Mixing checkpoints from different quant tools (e.g. an AWQ/GPTQ-style checkpoint fed to the Comfy W4A8 path), re-saving safetensors with dtype conversion, or using a model variant exported with a newer/different format version.

Related errors

• Comfy W4A8 layer {prefix!r} needs F32 channel scales, got {c
• Comfy W4A8 layer {prefix!r} needs a 2D packed weight, got {w
• Comfy W4A8 layer {prefix!r} needs an F32[16] codebook
• Comfy W4A4 layer {prefix!r} needs I8 packed weights and F32
• Type must match: {self.a_dtype} != {self.b_dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e11f0d740ac140eb. Report an issue: GitHub.