{"record":{"id":"6e3111adee3d184b","repo":"sgl-project/sglang","slug":"comfy-tensorwise-int8-layer-prefix-r-needs-a-2d","errorCode":null,"errorMessage":"Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, got {weight_shape}","messagePattern":"Comfy tensorwise INT8 layer (.+?) needs a 2D weight, got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/multimodal_gen/runtime/utils/quantization_utils.py","lineNumber":294,"sourceCode":"            pre_quant_scale_key = f\"{prefix}.pre_quant_scale\"\n            marker[\"_has_pre_quant_scale\"] = pre_quant_scale_key in checkpoint_meta\n            if marker[\"_has_pre_quant_scale\"]:\n                pre_scale_dtype, pre_scale_shape = checkpoint_meta[pre_quant_scale_key]\n                if pre_scale_dtype not in (\"BF16\", \"F16\", \"F32\") or (\n                    pre_scale_shape != (logical_input_size,)\n                ):\n                    raise ValueError(\n                        f\"Comfy NVFP4 layer {prefix!r} has an incompatible \"\n                        f\"pre_quant_scale: {pre_scale_dtype}{pre_scale_shape}\"\n                    )\n            continue\n        if marker_format != \"int8_tensorwise\":\n            continue\n        weight_dtype, weight_shape = checkpoint_meta[f\"{prefix}.weight\"]\n        scale_dtype, scale_shape = checkpoint_meta[f\"{prefix}.weight_scale\"]\n        if weight_dtype == \"I8\" and scale_dtype == \"F32\" and scale_shape == ():\n            if len(weight_shape) != 2:\n                raise ValueError(\n                    f\"Comfy tensorwise INT8 layer {prefix!r} needs a 2D weight, \"\n                    f\"got {weight_shape}\"\n                )\n            marker[\"_is_tensorwise_scalar\"] = True\n            continue\n        if weight_dtype != \"I8\" or scale_dtype != \"F32\":\n            raise ValueError(\n                f\"Comfy INT8 layer {prefix!r} needs I8 weights and F32 scales, \"\n                f\"got {weight_dtype} and {scale_dtype}\"\n            )\n        if len(weight_shape) != 2 or scale_shape != (weight_shape[0], 1):\n            raise ValueError(\n                f\"Comfy INT8 layer {prefix!r} has incompatible weight/scale \"\n                f\"shapes: {weight_shape} and {scale_shape}\"\n            )\n        marker[\"_is_rowwise\"] = True\n\n    mapped_markers: dict[str, dict[str, Any]] = {}","sourceCodeStart":276,"sourceCodeEnd":312,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/utils/quantization_utils.py#L276-L312","documentation":"In the int8_tensorwise fast path (weight I8, scale scalar F32), the checker requires the weight to be a 2D matrix before tagging the marker `_is_tensorwise_scalar`. Non-2D weights (conv kernels, biases mistakenly named .weight) can't use the tensorwise GEMM path, so they're rejected.","triggerScenarios":"`inspect_comfy_quant_markers` on an int8_tensorwise marker where `{prefix}.weight` is I8, `{prefix}.weight_scale` is a scalar F32, but the weight tensor is not 2D (e.g. a 4D conv weight or 1D tensor).","commonSituations":"A conv layer exported under int8_tensorwise without flattening to 2D; a marker accidentally applied to a non-linear module; exporter bugs that keep conv weights in kernel layout.","solutions":["Check weight_shape rank for the failing prefix","Re-export flattening conv weights to (out, in*kh*kw) or exclude conv layers from int8_tensorwise quantization","Fix the marker format field if the layer was actually quantized with a different scheme","Verify no non-weight tensor collides with the `{prefix}.weight` key"],"exampleFix":"# before: weight shape (512, 256, 1, 1)\n# after (flattened): weight shape (512, 256)","handlingStrategy":"validation","validationCode":"_, w_shape = checkpoint_meta[f\"{prefix}.weight\"]\nsd, s_shape = checkpoint_meta[f\"{prefix}.weight_scale\"]\nif checkpoint_meta[f\"{prefix}.weight\"][0] == \"I8\" and sd == \"F32\" and s_shape == ():\n    assert len(w_shape) == 2, f\"tensorwise INT8 needs 2D weight, got {w_shape}\"","typeGuard":"def is_2d_int8_weight(meta: dict, prefix: str) -> bool:\n    return len(meta[f\"{prefix}.weight\"][1]) == 2","tryCatchPattern":null,"preventionTips":["Flatten conv kernels to 2D or skip int8_tensorwise for conv layers","Validate weight rank in the export self-test","Keep marker assignment per-module-type explicit in the export config"],"tags":["quantization","int8","tensorwise","rank-mismatch","checkpoint-validation"],"backgroundTag":"tensor-shape-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}