{"record":{"id":"f0a16664279a4a6a","repo":"sgl-project/sglang","slug":"comfy-nvfp4-layer-prefix-r-needs-a-scalar-f32-we","errorCode":null,"errorMessage":"Comfy NVFP4 layer {prefix!r} needs a scalar F32 weight_scale_2, got {scale_2_dtype}{scale_2_shape}","messagePattern":"Comfy NVFP4 layer (.+?) needs a scalar F32 weight_scale_2, got (.+?)(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/utils/quantization_utils.py","lineNumber":260,"sourceCode":"                )\n            if logical_input_size % 64 or logical_input_size % convrot_group_size:\n                raise ValueError(\n                    f\"Comfy W4A4 layer {prefix!r} has input size \"\n                    f\"{logical_input_size}, incompatible with quant_group_size=64 \"\n                    f\"and convrot_groupsize={convrot_group_size}\"\n                )\n            continue\n        if marker_format == \"nvfp4\":\n            weight_dtype, weight_shape = checkpoint_meta[f\"{prefix}.weight\"]\n            scale_dtype, scale_shape = checkpoint_meta[f\"{prefix}.weight_scale\"]\n            scale_2_dtype, scale_2_shape = checkpoint_meta[f\"{prefix}.weight_scale_2\"]\n            if weight_dtype != \"U8\" or scale_dtype != \"F8_E4M3\":\n                raise ValueError(\n                    f\"Comfy NVFP4 layer {prefix!r} needs U8 packed weights and \"\n                    f\"FP8 block scales, got {weight_dtype} and {scale_dtype}\"\n                )\n            if scale_2_dtype != \"F32\" or scale_2_shape not in ((), (1,)):\n                raise ValueError(\n                    f\"Comfy NVFP4 layer {prefix!r} needs a scalar F32 \"\n                    f\"weight_scale_2, got {scale_2_dtype}{scale_2_shape}\"\n                )\n            if len(weight_shape) != 2:\n                raise ValueError(\n                    f\"Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, \"\n                    f\"got {weight_shape}\"\n                )\n            logical_input_size = weight_shape[1] * 2\n            expected_scale_shape = (weight_shape[0], logical_input_size // 16)\n            if logical_input_size % 16 or scale_shape != expected_scale_shape:\n                raise ValueError(\n                    f\"Comfy NVFP4 layer {prefix!r} has incompatible weight/scale \"\n                    f\"shapes: {weight_shape} and {scale_shape}\"\n                )\n            pre_quant_scale_key = f\"{prefix}.pre_quant_scale\"\n            marker[\"_has_pre_quant_scale\"] = pre_quant_scale_key in checkpoint_meta\n            if marker[\"_has_pre_quant_scale\"]:","sourceCodeStart":242,"sourceCodeEnd":278,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/utils/quantization_utils.py#L242-L278","documentation":"NVFP4 quantization uses a two-level scale: per-block FP8 scales plus one global scalar (`weight_scale_2`) that must be F32 and either a 0-d scalar or shape (1,). The checker raises when `weight_scale_2` has any other dtype (e.g. BF16) or a non-scalar shape such as per-channel (out,) or (1,1).","triggerScenarios":"`inspect_comfy_quant_markers` on an nvfp4 layer whose `{prefix}.weight_scale_2` is non-scalar (per-row vector) or stored in BF16/F16 instead of F32.","commonSituations":"An exporter variant that emits per-output-channel second-level scales instead of a single global scalar; dtype downcast during checkpoint conversion; hand-merged or sharded checkpoints where scale_2 got reshaped.","solutions":["Check the actual shape/dtype of `{prefix}.weight_scale_2` in the safetensors header","Re-export so weight_scale_2 is a single F32 scalar (shape () or (1,))","If the tensor is per-channel by mistake, collapse it to its global value or re-quantize","Upgrade SGLang if per-channel second scales become supported"],"exampleFix":"# before: weight_scale_2 dtype BF16 shape (3072,)\n# after: weight_scale_2 dtype F32 shape () or (1,)","handlingStrategy":"validation","validationCode":"d2, s2 = checkpoint_meta[f\"{prefix}.weight_scale_2\"]\nassert d2 == \"F32\" and s2 in ((), (1,)), f\"weight_scale_2 must be scalar F32, got {d2}{s2}\"","typeGuard":"def is_scalar_f32_scale_2(meta: dict, prefix: str) -> bool:\n    d, s = meta[f\"{prefix}.weight_scale_2\"]\n    return d == \"F32\" and s in ((), (1,))","tryCatchPattern":null,"preventionTips":["Assert weight_scale_2 is scalar F32 before invoking the inspector","When merging/sharding checkpoints, keep weight_scale_2 unsharded as a single global scalar","Log scale_2 shape at export time to catch per-channel regressions early"],"tags":["quantization","nvfp4","scalar-scale","dtype-mismatch","checkpoint-validation"],"backgroundTag":"quantization-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}