sgl-project/sglang · error · ValueError

Comfy INT8 layer {prefix!r} needs I8 weights and F32 scales,

Error message

Comfy INT8 layer {prefix!r} needs I8 weights and F32 scales, got {weight_dtype} and {scale_dtype}

What it means

For int8 layers that are not tensorwise-scalar (i.e. rowwise quantization), the checker requires weight dtype I8 and per-row scale dtype F32. If either dtype differs — e.g. weights saved as BF16/F16 while the marker claims int8_tensorwise, or scales stored as F16/BF64 — the layer is rejected because the INT8 kernels can't consume it.

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/quantization_utils.py:301

                    raise ValueError(
                        f"Comfy NVFP4 layer {prefix!r} has an incompatible "
                        f"pre_quant_scale: {pre_scale_dtype}{pre_scale_shape}"
                    )
            continue
        if marker_format != "int8_tensorwise":
            continue
        weight_dtype, weight_shape = checkpoint_meta[f"{prefix}.weight"]
        scale_dtype, scale_shape = checkpoint_meta[f"{prefix}.weight_scale"]
        if weight_dtype == "I8" and scale_dtype == "F32" and scale_shape == ():
            if len(weight_shape) != 2:
                raise ValueError(
                    f"Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, "
                    f"got {weight_shape}"
                )
            marker["_is_tensorwise_scalar"] = True
            continue
        if weight_dtype != "I8" or scale_dtype != "F32":
            raise ValueError(
                f"Comfy INT8 layer {prefix!r} needs I8 weights and F32 scales, "
                f"got {weight_dtype} and {scale_dtype}"
            )
        if len(weight_shape) != 2 or scale_shape != (weight_shape[0], 1):
            raise ValueError(
                f"Comfy INT8 layer {prefix!r} has incompatible weight/scale "
                f"shapes: {weight_shape} and {scale_shape}"
            )
        marker["_is_rowwise"] = True

    mapped_markers: dict[str, dict[str, Any]] = {}
    for prefix, marker in raw_markers.items():
        mapped_prefix = param_name_mapper(prefix) if param_name_mapper else prefix
        if mapped_prefix in mapped_markers:
            raise ValueError(
                f"Comfy markers collide after parameter mapping at {mapped_prefix!r}"
            )
        mapped_markers[mapped_prefix] = marker

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect dtypes of `{prefix}.weight` and `{prefix}.weight_scale` in the safetensors header
2. If the weight is BF16, remove the int8 marker for that layer or actually quantize it to I8
3. Re-export ensuring scales are saved as F32
4. If the format marker is wrong, correct it to the layer's real format

Example fix

# before: weight dtype BF16, weight_scale dtype F16
# after: weight dtype I8, weight_scale dtype F32

wd, _ = checkpoint_meta[f"{prefix}.weight"]
sd, _ = checkpoint_meta[f"{prefix}.weight_scale"]
assert wd == "I8" and sd == "F32", f"{prefix}: INT8 needs I8/F32, got {wd}/{sd}"

def is_int8_f32(meta: dict, prefix: str) -> bool:
    return (
        meta[f"{prefix}.weight"][0] == "I8"
        and meta[f"{prefix}.weight_scale"][0] == "F32"
    )

When it happens

Trigger: `inspect_comfy_quant_markers` on an int8_tensorwise marker where `{prefix}.weight` is not I8 or `{prefix}.weight_scale` is not F32, and the tensorwise-scalar shortcut (scalar scale) didn't match. Typical when a layer's weights were left unquantized (BF16) but the marker still says int8.

Common situations: Mixed-precision exports where some layers fall back to BF16 but keep int8 markers; dtype re-casting during checkpoint conversion (F32 scales downcast to F16); marker format fields misassigned during a batch export.

Related errors

• Comfy NVFP4 layer {prefix!r} needs U8 packed weights and FP8
• Comfy NVFP4 layer {prefix!r} needs a scalar F32 weight_scale
• Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, go
• Comfy INT8 layer {prefix!r} has incompatible weight/scale sh
• Type must match: {self.a_dtype} != {self.b_dtype}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/4135deb36b3401b4. Report an issue: GitHub.