sgl-project/sglang · error · ValueError

Comfy INT8 layer {prefix!r} has incompatible weight/scale sh

Error message

Comfy INT8 layer {prefix!r} has incompatible weight/scale shapes: {weight_shape} and {scale_shape}

What it means

For rowwise int8 layers (I8 weight, F32 non-scalar scale), the checker requires a 2D weight and scale shape exactly (out_features, 1) — one scale per output row. Any other combination (non-2D weight, per-column scale, scalar scale with wrong path, mismatched first dim) means dequantization would misalign.

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/quantization_utils.py:306

        if marker_format != "int8_tensorwise":
            continue
        weight_dtype, weight_shape = checkpoint_meta[f"{prefix}.weight"]
        scale_dtype, scale_shape = checkpoint_meta[f"{prefix}.weight_scale"]
        if weight_dtype == "I8" and scale_dtype == "F32" and scale_shape == ():
            if len(weight_shape) != 2:
                raise ValueError(
                    f"Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, "
                    f"got {weight_shape}"
                )
            marker["_is_tensorwise_scalar"] = True
            continue
        if weight_dtype != "I8" or scale_dtype != "F32":
            raise ValueError(
                f"Comfy INT8 layer {prefix!r} needs I8 weights and F32 scales, "
                f"got {weight_dtype} and {scale_dtype}"
            )
        if len(weight_shape) != 2 or scale_shape != (weight_shape[0], 1):
            raise ValueError(
                f"Comfy INT8 layer {prefix!r} has incompatible weight/scale "
                f"shapes: {weight_shape} and {scale_shape}"
            )
        marker["_is_rowwise"] = True

    mapped_markers: dict[str, dict[str, Any]] = {}
    for prefix, marker in raw_markers.items():
        mapped_prefix = param_name_mapper(prefix) if param_name_mapper else prefix
        if mapped_prefix in mapped_markers:
            raise ValueError(
                f"Comfy markers collide after parameter mapping at {mapped_prefix!r}"
            )
        mapped_markers[mapped_prefix] = marker
    return mapped_markers


def resolve_comfy_checkpoint_quantization(
    layer_markers: dict[str, dict[str, Any]],

View on GitHub (pinned to 0132848349)

Solutions

1. Print weight_shape and scale_shape for the failing prefix; expected scale is (weight_shape[0], 1)
2. Re-export with per-output-row scales, or transpose/reformat the scale tensor to match
3. If the weight isn't 2D, flatten conv kernels or exclude the layer from int8 quantization
4. Ensure the exporter's row/column convention matches SGLang's expected rowwise layout

Example fix

# before: weight (3072, 4096), scale (1, 3072)  # columnwise
# after: scale (3072, 1)  # one F32 scale per output row

_, w_shape = checkpoint_meta[f"{prefix}.weight"]
_, s_shape = checkpoint_meta[f"{prefix}.weight_scale"]
assert len(w_shape) == 2 and s_shape == (w_shape[0], 1), (
    f"rowwise INT8 scale must be {(w_shape[0], 1)}, got {s_shape}"
)

def rowwise_int8_shapes_ok(w_shape, s_shape) -> bool:
    return len(w_shape) == 2 and s_shape == (w_shape[0], 1)

When it happens

Trigger: `inspect_comfy_quant_markers` on an int8_tensorwise marker where weight is I8, scale is F32 non-scalar, but scale_shape != (weight_shape[0], 1) or the weight isn't 2D — e.g. scale saved as (1, in_features) (columnwise) or (out,) flattened.

Common situations: Exporter emitting columnwise instead of rowwise scales; transposed weights between exporter versions; conv weights left 4D; scale flattened to 1D during conversion.

Related errors

• Comfy W4A4 layer {prefix!r} has input size {logical_input_si
• Comfy NVFP4 layer {prefix!r} has incompatible weight/scale s
• Comfy NVFP4 layer {prefix!r} has an incompatible pre_quant_s
• Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, go
• Comfy INT8 layer {prefix!r} needs I8 weights and F32 scales,

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b9825390f5c3ae68. Report an issue: GitHub.