{"record":{"id":"a7c89d40bb6295b0","repo":"sgl-project/sglang","slug":"comfy-nvfp4-layer-prefix-r-has-an-incompatible-p","errorCode":null,"errorMessage":"Comfy NVFP4 layer {prefix!r} has an incompatible pre_quant_scale: {pre_scale_dtype}{pre_scale_shape}","messagePattern":"Comfy NVFP4 layer (.+?) has an incompatible pre_quant_scale: (.+?)(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/utils/quantization_utils.py","lineNumber":283,"sourceCode":"                raise ValueError(\n                    f\"Comfy NVFP4 layer {prefix!r} needs a 2D packed weight, \"\n                    f\"got {weight_shape}\"\n                )\n            logical_input_size = weight_shape[1] * 2\n            expected_scale_shape = (weight_shape[0], logical_input_size // 16)\n            if logical_input_size % 16 or scale_shape != expected_scale_shape:\n                raise ValueError(\n                    f\"Comfy NVFP4 layer {prefix!r} has incompatible weight/scale \"\n                    f\"shapes: {weight_shape} and {scale_shape}\"\n                )\n            pre_quant_scale_key = f\"{prefix}.pre_quant_scale\"\n            marker[\"_has_pre_quant_scale\"] = pre_quant_scale_key in checkpoint_meta\n            if marker[\"_has_pre_quant_scale\"]:\n                pre_scale_dtype, pre_scale_shape = checkpoint_meta[pre_quant_scale_key]\n                if pre_scale_dtype not in (\"BF16\", \"F16\", \"F32\") or (\n                    pre_scale_shape != (logical_input_size,)\n                ):\n                    raise ValueError(\n                        f\"Comfy NVFP4 layer {prefix!r} has an incompatible \"\n                        f\"pre_quant_scale: {pre_scale_dtype}{pre_scale_shape}\"\n                    )\n            continue\n        if marker_format != \"int8_tensorwise\":\n            continue\n        weight_dtype, weight_shape = checkpoint_meta[f\"{prefix}.weight\"]\n        scale_dtype, scale_shape = checkpoint_meta[f\"{prefix}.weight_scale\"]\n        if weight_dtype == \"I8\" and scale_dtype == \"F32\" and scale_shape == ():\n            if len(weight_shape) != 2:\n                raise ValueError(\n                    f\"Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, \"\n                    f\"got {weight_shape}\"\n                )\n            marker[\"_is_tensorwise_scalar\"] = True\n            continue\n        if weight_dtype != \"I8\" or scale_dtype != \"F32\":\n            raise ValueError(","sourceCodeStart":265,"sourceCodeEnd":301,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/utils/quantization_utils.py#L265-L301","documentation":"When an nvfp4 layer optionally carries a `pre_quant_scale` (input activation scaling before quantization), the checker requires it to be BF16/F16/F32 with shape exactly (logical_input_size,) — one scalar per input feature. Any other dtype (e.g. F8) or shape (per-tensor scalar, wrong length, 2D) is rejected because the activation-quant kernel indexes it per channel.","triggerScenarios":"`inspect_comfy_quant_markers` on an nvfp4 layer that has `{prefix}.pre_quant_scale` in the checkpoint, where the tensor's shape is not (in_features,) — e.g. () (a global scalar), (1,), or (out_features,) — or its dtype is FP8/BF8.","commonSituations":"Exporter emitting a single global activation scale instead of per-channel; checkpoint converted between SmoothQuant-style and per-channel recipes; input features changed after quantization (model resize/patching), making the length mismatch.","solutions":["Inspect pre_quant_scale dtype/shape for the failing prefix and compare with weight_shape[1]*2","Re-export with per-input-channel pre_quant_scale of shape (in_features,) in BF16/F16/F32","If your recipe uses a global activation scale, re-quantize without pre_quant_scale or with per-channel scaling","If in_features changed post-export, re-quantize against the current architecture"],"exampleFix":"# before: pre_quant_scale dtype F32 shape (1,)\n# after: pre_quant_scale dtype F32 shape (4096,)","handlingStrategy":"validation","validationCode":"key = f\"{prefix}.pre_quant_scale\"\nif key in checkpoint_meta:\n    d, s = checkpoint_meta[key]\n    logical_in = checkpoint_meta[f\"{prefix}.weight\"][1][1] * 2\n    assert d in (\"BF16\", \"F16\", \"F32\") and s == (logical_in,), f\"bad pre_quant_scale {d}{s}\"","typeGuard":"def pre_quant_scale_ok(meta: dict, prefix: str) -> bool:\n    key = f\"{prefix}.pre_quant_scale\"\n    if key not in meta:\n        return True\n    d, s = meta[key]\n    return d in (\"BF16\", \"F16\", \"F32\") and s == (meta[f\"{prefix}.weight\"][1][1] * 2,)","tryCatchPattern":null,"preventionTips":["Emit per-input-channel pre_quant_scale of length in_features at export","Avoid global activation-scale recipes when targeting SGLang NVFP4","Re-quantize rather than editing architectures after export"],"tags":["quantization","nvfp4","pre-quant-scale","shape-mismatch","checkpoint-validation"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}