sgl-project/sglang · error · ValueError

Comfy NVFP4 layer {prefix!r} needs a scalar F32 weight_scale

Error message

Comfy NVFP4 layer {prefix!r} needs a scalar F32 weight_scale_2, got {scale_2_dtype}{scale_2_shape}

What it means

NVFP4 quantization uses a two-level scale: per-block FP8 scales plus one global scalar (`weight_scale_2`) that must be F32 and either a 0-d scalar or shape (1,). The checker raises when `weight_scale_2` has any other dtype (e.g. BF16) or a non-scalar shape such as per-channel (out,) or (1,1).

Source

Thrown at python/sglang/multimodal_gen/runtime/utils/quantization_utils.py:260

                )
            if logical_input_size % 64 or logical_input_size % convrot_group_size:
                raise ValueError(
                    f"Comfy W4A4 layer {prefix!r} has input size "
                    f"{logical_input_size}, incompatible with quant_group_size=64 "
                    f"and convrot_groupsize={convrot_group_size}"
                )
            continue
        if marker_format == "nvfp4":
            weight_dtype, weight_shape = checkpoint_meta[f"{prefix}.weight"]
            scale_dtype, scale_shape = checkpoint_meta[f"{prefix}.weight_scale"]
            scale_2_dtype, scale_2_shape = checkpoint_meta[f"{prefix}.weight_scale_2"]
            if weight_dtype != "U8" or scale_dtype != "F8_E4M3":
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs U8 packed weights and "
                    f"FP8 block scales, got {weight_dtype} and {scale_dtype}"
                )
            if scale_2_dtype != "F32" or scale_2_shape not in ((), (1,)):
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs a scalar F32 "
                    f"weight_scale_2, got {scale_2_dtype}{scale_2_shape}"
                )
            if len(weight_shape) != 2:
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, "
                    f"got {weight_shape}"
                )
            logical_input_size = weight_shape[1] * 2
            expected_scale_shape = (weight_shape[0], logical_input_size // 16)
            if logical_input_size % 16 or scale_shape != expected_scale_shape:
                raise ValueError(
                    f"Comfy NVFP4 layer {prefix!r} has incompatible weight/scale "
                    f"shapes: {weight_shape} and {scale_shape}"
                )
            pre_quant_scale_key = f"{prefix}.pre_quant_scale"
            marker["_has_pre_quant_scale"] = pre_quant_scale_key in checkpoint_meta
            if marker["_has_pre_quant_scale"]:

View on GitHub (pinned to 0132848349)

Solutions

1. Check the actual shape/dtype of `{prefix}.weight_scale_2` in the safetensors header
2. Re-export so weight_scale_2 is a single F32 scalar (shape () or (1,))
3. If the tensor is per-channel by mistake, collapse it to its global value or re-quantize
4. Upgrade SGLang if per-channel second scales become supported

Example fix

# before: weight_scale_2 dtype BF16 shape (3072,)
# after: weight_scale_2 dtype F32 shape () or (1,)

d2, s2 = checkpoint_meta[f"{prefix}.weight_scale_2"]
assert d2 == "F32" and s2 in ((), (1,)), f"weight_scale_2 must be scalar F32, got {d2}{s2}"

def is_scalar_f32_scale_2(meta: dict, prefix: str) -> bool:
    d, s = meta[f"{prefix}.weight_scale_2"]
    return d == "F32" and s in ((), (1,))

When it happens

Trigger: `inspect_comfy_quant_markers` on an nvfp4 layer whose `{prefix}.weight_scale_2` is non-scalar (per-row vector) or stored in BF16/F16 instead of F32.

Common situations: An exporter variant that emits per-output-channel second-level scales instead of a single global scalar; dtype downcast during checkpoint conversion; hand-merged or sharded checkpoints where scale_2 got reshaped.

Related errors

• Comfy NVFP4 layer {prefix!r} needs U8 packed weights and FP8
• Type must match: {self.a_dtype} != {self.b_dtype}
• Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, got {
• Comfy NVFP4 layer {prefix!r} has incompatible weight/scale s
• Comfy NVFP4 layer {prefix!r} has an incompatible pre_quant_s

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/f0a16664279a4a6a. Report an issue: GitHub.