{"record":{"id":"4135deb36b3401b4","repo":"sgl-project/sglang","slug":"comfy-int8-layer-prefix-r-needs-i8-weights-and-f","errorCode":null,"errorMessage":"Comfy INT8 layer {prefix!r} needs I8 weights and F32 scales, got {weight_dtype} and {scale_dtype}","messagePattern":"Comfy INT8 layer (.+?) needs I8 weights and F32 scales, got (.+?) and (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/utils/quantization_utils.py","lineNumber":301,"sourceCode":"                    raise ValueError(\n                        f\"Comfy NVFP4 layer {prefix!r} has an incompatible \"\n                        f\"pre_quant_scale: {pre_scale_dtype}{pre_scale_shape}\"\n                    )\n            continue\n        if marker_format != \"int8_tensorwise\":\n            continue\n        weight_dtype, weight_shape = checkpoint_meta[f\"{prefix}.weight\"]\n        scale_dtype, scale_shape = checkpoint_meta[f\"{prefix}.weight_scale\"]\n        if weight_dtype == \"I8\" and scale_dtype == \"F32\" and scale_shape == ():\n            if len(weight_shape) != 2:\n                raise ValueError(\n                    f\"Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, \"\n                    f\"got {weight_shape}\"\n                )\n            marker[\"_is_tensorwise_scalar\"] = True\n            continue\n        if weight_dtype != \"I8\" or scale_dtype != \"F32\":\n            raise ValueError(\n                f\"Comfy INT8 layer {prefix!r} needs I8 weights and F32 scales, \"\n                f\"got {weight_dtype} and {scale_dtype}\"\n            )\n        if len(weight_shape) != 2 or scale_shape != (weight_shape[0], 1):\n            raise ValueError(\n                f\"Comfy INT8 layer {prefix!r} has incompatible weight/scale \"\n                f\"shapes: {weight_shape} and {scale_shape}\"\n            )\n        marker[\"_is_rowwise\"] = True\n\n    mapped_markers: dict[str, dict[str, Any]] = {}\n    for prefix, marker in raw_markers.items():\n        mapped_prefix = param_name_mapper(prefix) if param_name_mapper else prefix\n        if mapped_prefix in mapped_markers:\n            raise ValueError(\n                f\"Comfy markers collide after parameter mapping at {mapped_prefix!r}\"\n            )\n        mapped_markers[mapped_prefix] = marker","sourceCodeStart":283,"sourceCodeEnd":319,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/utils/quantization_utils.py#L283-L319","documentation":"For int8 layers that are not tensorwise-scalar (i.e. rowwise quantization), the checker requires weight dtype I8 and per-row scale dtype F32. If either dtype differs — e.g. weights saved as BF16/F16 while the marker claims int8_tensorwise, or scales stored as F16/BF64 — the layer is rejected because the INT8 kernels can't consume it.","triggerScenarios":"`inspect_comfy_quant_markers` on an int8_tensorwise marker where `{prefix}.weight` is not I8 or `{prefix}.weight_scale` is not F32, and the tensorwise-scalar shortcut (scalar scale) didn't match. Typical when a layer's weights were left unquantized (BF16) but the marker still says int8.","commonSituations":"Mixed-precision exports where some layers fall back to BF16 but keep int8 markers; dtype re-casting during checkpoint conversion (F32 scales downcast to F16); marker format fields misassigned during a batch export.","solutions":["Inspect dtypes of `{prefix}.weight` and `{prefix}.weight_scale` in the safetensors header","If the weight is BF16, remove the int8 marker for that layer or actually quantize it to I8","Re-export ensuring scales are saved as F32","If the format marker is wrong, correct it to the layer's real format"],"exampleFix":"# before: weight dtype BF16, weight_scale dtype F16\n# after: weight dtype I8, weight_scale dtype F32","handlingStrategy":"validation","validationCode":"wd, _ = checkpoint_meta[f\"{prefix}.weight\"]\nsd, _ = checkpoint_meta[f\"{prefix}.weight_scale\"]\nassert wd == \"I8\" and sd == \"F32\", f\"{prefix}: INT8 needs I8/F32, got {wd}/{sd}\"","typeGuard":"def is_int8_f32(meta: dict, prefix: str) -> bool:\n    return (\n        meta[f\"{prefix}.weight\"][0] == \"I8\"\n        and meta[f\"{prefix}.weight_scale\"][0] == \"F32\"\n    )","tryCatchPattern":null,"preventionTips":["Don't leave int8 markers on layers that fell back to BF16 during export","Save INT8 scales as F32; avoid F16 downcasting in conversion pipelines","Run a dtype audit over all quantized prefixes before deployment"],"tags":["quantization","int8","dtype-mismatch","checkpoint-validation"],"backgroundTag":"quantization-dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}