{"record":{"id":"848a863275a7f54b","repo":"sgl-project/sglang","slug":"unsupported-quantized-embedding-marker-for-prefix","errorCode":null,"errorMessage":"Unsupported quantized embedding marker for {prefix!r}: {marker}","messagePattern":"Unsupported quantized embedding marker for (.+?): (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"critical","filePath":"python/sglang/multimodal_gen/runtime/layers/quantization/comfy_nvfp4.py","lineNumber":227,"sourceCode":"\n    @classmethod\n    def from_config(cls, config: dict[str, Any]) -> ComfyNvfp4Config:\n        raise ValueError(\n            \"comfy_nvfp4 is inferred from per-layer checkpoint metadata; \"\n            \"it is not an online quantization method\"\n        )\n\n    def get_quant_method(\n        self, layer: nn.Module, prefix: str\n    ) -> QuantizeMethodBase | None:\n        marker = self.layer_markers.get(prefix)\n        if isinstance(layer, VocabParallelEmbedding):\n            if marker is None:\n                return None\n            if marker.get(\"format\") != \"int8_tensorwise\" or not marker.get(\n                \"_is_rowwise\"\n            ):\n                raise ValueError(\n                    f\"Unsupported quantized embedding marker for {prefix!r}: {marker}\"\n                )\n            self.selected.append(prefix)\n            return ComfyRowwiseInt8EmbeddingMethod()\n        if not isinstance(layer, LinearBase):\n            return None\n        if marker is None:\n            return UnquantizedLinearMethod()\n        if marker.get(\"format\") != \"nvfp4\":\n            raise ValueError(f\"Unsupported quantized linear marker for {prefix!r}\")\n        self.selected.append(prefix)\n        return ComfyFullPrecisionNvfp4LinearMethod(\n            self,\n            has_pre_quant_scale=bool(marker.get(\"_has_pre_quant_scale\")),\n        )\n\n    def quantizes_embedding(self, prefix: str) -> bool:\n        marker = self.layer_markers.get(prefix)","sourceCodeStart":209,"sourceCodeEnd":245,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/multimodal_gen/runtime/layers/quantization/comfy_nvfp4.py#L209-L245","documentation":"Raised by ComfyNvfp4Config.get_quant_method when a VocabParallelEmbedding layer has a checkpoint quantization marker whose format is not 'int8_tensorwise' with _is_rowwise=true. The Comfy nvfp4 quant config only supports rowwise int8-quantized embeddings; any other marker format on an embedding layer is rejected at model-load time.","triggerScenarios":"Loading a ComfyUI-exported checkpoint whose embedding layer marker has format != 'int8_tensorwise' (e.g. 'nvfp4' or 'fp8') or is missing _is_rowwise, while the model config selects the comfy_nvfp4 quantization method.","commonSituations":"Exporting a model from ComfyUI with mixed quantization recipes (embedding quantized with a different scheme than linear layers), or hand-editing layer_markers in the checkpoint config.","solutions":["Re-export/quantize the model so embedding markers use format 'int8_tensorwise' with _is_rowwise: true","Remove the embedding entry from layer_markers so marker is None and the embedding stays unquantized","Move the model to a quantization config that supports the embedding format actually present"],"exampleFix":"// before\nlayer_markers = {\"model.embed_tokens\": {\"format\": \"nvfp4\"}}\n// after\nlayer_markers = {\"model.embed_tokens\": {\"format\": \"int8_tensorwise\", \"_is_rowwise\": True}}","handlingStrategy":"validation","validationCode":"marker = layer_markers.get(prefix)\nif isinstance(layer, VocabParallelEmbedding) and marker is not None:\n    assert marker.get(\"format\") == \"int8_tensorwise\" and marker.get(\"_is_rowwise\"), marker","typeGuard":"def is_comfy_rowwise_int8_marker(m: dict) -> bool:\n    return m.get(\"format\") == \"int8_tensorwise\" and bool(m.get(\"_is_rowwise\"))","tryCatchPattern":null,"preventionTips":["Validate all layer_markers against the target quant config's accepted formats before loading","Keep exporter and sglang runtime versions in lockstep"],"tags":["quantization","checkpoint","embedding","model-load"],"backgroundTag":"unsupported-quantization-config","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}