sgl-project/sglang · error · ValueError

Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, go

Error message

Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, got {weight_shape}

What it means

In the int8_tensorwise fast path (weight I8, scale scalar F32), the checker requires the weight to be a 2D matrix before tagging the marker `_is_tensorwise_scalar`. Non-2D weights (conv kernels, biases mistakenly named .weight) can't use the tensorwise GEMM path, so they're rejected.

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/quantization_utils.py:294

            pre_quant_scale_key = f"{prefix}.pre_quant_scale"
            marker["_has_pre_quant_scale"] = pre_quant_scale_key in checkpoint_meta
            if marker["_has_pre_quant_scale"]:
                pre_scale_dtype, pre_scale_shape = checkpoint_meta[pre_quant_scale_key]
                if pre_scale_dtype not in ("BF16", "F16", "F32") or (
                    pre_scale_shape != (logical_input_size,)
                ):
                    raise ValueError(
                        f"Comfy NVFP4 layer {prefix!r} has an incompatible "
                        f"pre_quant_scale: {pre_scale_dtype}{pre_scale_shape}"
                    )
            continue
        if marker_format != "int8_tensorwise":
            continue
        weight_dtype, weight_shape = checkpoint_meta[f"{prefix}.weight"]
        scale_dtype, scale_shape = checkpoint_meta[f"{prefix}.weight_scale"]
        if weight_dtype == "I8" and scale_dtype == "F32" and scale_shape == ():
            if len(weight_shape) != 2:
                raise ValueError(
                    f"Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, "
                    f"got {weight_shape}"
                )
            marker["_is_tensorwise_scalar"] = True
            continue
        if weight_dtype != "I8" or scale_dtype != "F32":
            raise ValueError(
                f"Comfy INT8 layer {prefix!r} needs I8 weights and F32 scales, "
                f"got {weight_dtype} and {scale_dtype}"
            )
        if len(weight_shape) != 2 or scale_shape != (weight_shape[0], 1):
            raise ValueError(
                f"Comfy INT8 layer {prefix!r} has incompatible weight/scale "
                f"shapes: {weight_shape} and {scale_shape}"
            )
        marker["_is_rowwise"] = True

    mapped_markers: dict[str, dict[str, Any]] = {}

View on GitHub (pinned to 0132848349)

Solutions

Check weight_shape rank for the failing prefix
Re-export flattening conv weights to (out, in*kh*kw) or exclude conv layers from int8_tensorwise quantization
Fix the marker format field if the layer was actually quantized with a different scheme
Verify no non-weight tensor collides with the `{prefix}.weight` key

Example fix

# before: weight shape (512, 256, 1, 1)
# after (flattened): weight shape (512, 256)

Defensive patterns

Strategy: validation

Validate before calling

_, w_shape = checkpoint_meta[f"{prefix}.weight"]
sd, s_shape = checkpoint_meta[f"{prefix}.weight_scale"]
if checkpoint_meta[f"{prefix}.weight"][0] == "I8" and sd == "F32" and s_shape == ():
    assert len(w_shape) == 2, f"tensorwise INT8 needs 2D weight, got {w_shape}"

Type guard

def is_2d_int8_weight(meta: dict, prefix: str) -> bool:
    return len(meta[f"{prefix}.weight"][1]) == 2

Prevention

Flatten conv kernels to 2D or skip int8_tensorwise for conv layers
Validate weight rank in the export self-test
Keep marker assignment per-module-type explicit in the export config

When it happens

Trigger: `inspect_comfy_quant_markers` on an int8_tensorwise marker where `{prefix}.weight` is I8, `{prefix}.weight_scale` is a scalar F32, but the weight tensor is not 2D (e.g. a 4D conv weight or 1D tensor).

Common situations: A conv layer exported under int8_tensorwise without flattening to 2D; a marker accidentally applied to a non-linear module; exporter bugs that keep conv weights in kernel layout.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/6e3111adee3d184b. Report an issue: GitHub.