{"record":{"id":"81ecec96c92ac141","repo":"sgl-project/sglang","slug":"unsupported-params-dtype-params-dtype","errorCode":null,"errorMessage":"Unsupported params_dtype: {params_dtype}","messagePattern":"Unsupported params_dtype: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/modelslim/schemes/modelslim_w8a8_int8.py","lineNumber":101,"sourceCode":"                data=torch.empty(1, dtype=params_dtype),\n                weight_loader=weight_loader,\n            )\n            input_offset.ignore_warning = True\n            layer.register_parameter(\"input_offset\", input_offset)\n\n            quant_bias = ChannelQuantScaleParameter(\n                data=torch.empty(output_size_per_partition, dtype=torch.int32),\n                output_dim=0,\n                weight_loader=weight_loader,\n            )\n            layer.register_parameter(\"quant_bias\", quant_bias)\n\n            if params_dtype == torch.bfloat16:\n                deq_scale_dtype = torch.float32\n            elif params_dtype == torch.float16:\n                deq_scale_dtype = torch.int64\n            else:\n                raise ValueError(f\"Unsupported params_dtype: {params_dtype}\")\n            deq_scale = ChannelQuantScaleParameter(\n                data=torch.empty(output_size_per_partition, dtype=deq_scale_dtype),\n                output_dim=0,\n                weight_loader=weight_loader,\n            )\n            layer.register_parameter(\"deq_scale\", deq_scale)\n\n    def process_weights_after_loading(self, layer: torch.nn.Module):\n        self.kernel.process_weights_after_loading(layer)\n\n    def apply_weights(\n        self,\n        layer: torch.nn.Module,\n        x: torch.Tensor,\n        bias: Optional[torch.Tensor] = None,\n    ) -> torch.Tensor:\n        return self.kernel.apply(layer, x, bias)\n","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/modelslim/schemes/modelslim_w8a8_int8.py#L83-L119","documentation":"Raised while creating quantized weights for the ModelSlim W8A8 INT8 scheme. The dequantization scale tensor's dtype is derived from the layer's params_dtype: bfloat16 maps to float32 scales and float16 maps to int64 scales (NPU kernel requirement). Any other dtype — typically float32 — has no valid deq_scale representation and is rejected.","triggerScenarios":"Calling create_weights on the W8A8 INT8 quant method with layer.params_dtype set to torch.float32 (or any dtype other than torch.bfloat16 / torch.float16), e.g. by running the server with --dtype float32 on a ModelSlim-quantized NPU model.","commonSituations":"User forces --dtype float32 for debugging or CPU-side comparison; or a checkpoint/config carries float32 as the default dtype and it propagates into the quantized layer creation.","solutions":["Run with --dtype bfloat16 (or float16), matching the dtype the model was quantized/calibrated with","If float32 truly must be supported, patch create_weights to define a deq_scale dtype for it and verify the NPU kernel accepts it","Check the model config's torch_dtype field is bf16/fp16, not float32"],"exampleFix":"# before\npython -m sglang.launch_server --model ... --dtype float32\n# after\npython -m sglang.launch_server --model ... --dtype bfloat16","handlingStrategy":"validation","validationCode":"import torch\nassert params_dtype in (torch.bfloat16, torch.float16), (\n    f\"params_dtype {params_dtype} unsupported; use bf16/fp16\"\n)\nquant_method.create_weights(layer, params_dtype, ...)","typeGuard":"def is_supported_dtype(d: torch.dtype) -> TypeGuard[torch.dtype]:\n    return d in (torch.bfloat16, torch.float16)","tryCatchPattern":null,"preventionTips":["Always launch quantized NPU models with --dtype bfloat16 or float16","Validate checkpoint torch_dtype before loading it into a quant scheme"],"tags":["quantization","dtype","npu","modelslim","w8a8"],"backgroundTag":"unsupported-dtype","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}