sgl-project/sglang · error · ValueError

Comfy NVFP4 layer {prefix!r} has an incompatible pre_quant_s

Error message

Comfy NVFP4 layer {prefix!r} has an incompatible pre_quant_scale: {pre_scale_dtype}{pre_scale_shape}

What it means

When an nvfp4 layer optionally carries a `pre_quant_scale` (input activation scaling before quantization), the checker requires it to be BF16/F16/F32 with shape exactly (logical_input_size,) — one scalar per input feature. Any other dtype (e.g. F8) or shape (per-tensor scalar, wrong length, 2D) is rejected because the activation-quant kernel indexes it per channel.

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/quantization_utils.py:283

                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, "
                    f"got {weight_shape}"
                )
            logical_input_size = weight_shape[1] * 2
            expected_scale_shape = (weight_shape[0], logical_input_size // 16)
            if logical_input_size % 16 or scale_shape != expected_scale_shape:
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} has incompatible weight/scale "
                    f"shapes: {weight_shape} and {scale_shape}"
                )
            pre_quant_scale_key = f"{prefix}.pre_quant_scale"
            marker["_has_pre_quant_scale"] = pre_quant_scale_key in checkpoint_meta
            if marker["_has_pre_quant_scale"]:
                pre_scale_dtype, pre_scale_shape = checkpoint_meta[pre_quant_scale_key]
                if pre_scale_dtype not in ("BF16", "F16", "F32") or (
                    pre_scale_shape != (logical_input_size,)
                ):
                    raise ValueError(
                        f"Comfy NVFP4 layer {prefix!r} has an incompatible "
                        f"pre_quant_scale: {pre_scale_dtype}{pre_scale_shape}"
                    )
            continue
        if marker_format != "int8_tensorwise":
            continue
        weight_dtype, weight_shape = checkpoint_meta[f"{prefix}.weight"]
        scale_dtype, scale_shape = checkpoint_meta[f"{prefix}.weight_scale"]
        if weight_dtype == "I8" and scale_dtype == "F32" and scale_shape == ():
            if len(weight_shape) != 2:
                raise ValueError(
                    f"Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, "
                    f"got {weight_shape}"
                )
            marker["_is_tensorwise_scalar"] = True
            continue
        if weight_dtype != "I8" or scale_dtype != "F32":
            raise ValueError(

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect pre_quant_scale dtype/shape for the failing prefix and compare with weight_shape[1]*2
2. Re-export with per-input-channel pre_quant_scale of shape (in_features,) in BF16/F16/F32
3. If your recipe uses a global activation scale, re-quantize without pre_quant_scale or with per-channel scaling
4. If in_features changed post-export, re-quantize against the current architecture

Example fix

# before: pre_quant_scale dtype F32 shape (1,)
# after: pre_quant_scale dtype F32 shape (4096,)

key = f"{prefix}.pre_quant_scale"
if key in checkpoint_meta:
    d, s = checkpoint_meta[key]
    logical_in = checkpoint_meta[f"{prefix}.weight"][1][1] * 2
    assert d in ("BF16", "F16", "F32") and s == (logical_in,), f"bad pre_quant_scale {d}{s}"

def pre_quant_scale_ok(meta: dict, prefix: str) -> bool:
    key = f"{prefix}.pre_quant_scale"
    if key not in meta:
        return True
    d, s = meta[key]
    return d in ("BF16", "F16", "F32") and s == (meta[f"{prefix}.weight"][1][1] * 2,)

When it happens

Trigger: `inspect_comfy_quant_markers` on an nvfp4 layer that has `{prefix}.pre_quant_scale` in the checkpoint, where the tensor's shape is not (in_features,) — e.g. () (a global scalar), (1,), or (out_features,) — or its dtype is FP8/BF8.

Common situations: Exporter emitting a single global activation scale instead of per-channel; checkpoint converted between SmoothQuant-style and per-channel recipes; input features changed after quantization (model resize/patching), making the length mismatch.

Related errors

• Comfy NVFP4 layer {prefix!r} has incompatible weight/scale s
• Shape mismatch: A K={k}, B K={b.shape[1] * 2}
• Comfy W4A4 layer {prefix!r} has input size {logical_input_si
• Comfy NVFP4 layer {prefix!r} needs U8 packed weights and FP8
• Comfy NVFP4 layer {prefix!r} needs a scalar F32 weight_scale

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/a7c89d40bb6295b0. Report an issue: GitHub.