{"record":{"id":"b48cc5903a0fcf58","repo":"sgl-project/sglang","slug":"the-params-dtype-must-be-float16-but-got-params","errorCode":null,"errorMessage":"The params dtype must be float16, but got {params_dtype}","messagePattern":"The params dtype must be float16, but got (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/marlin_utils.py","lineNumber":755,"sourceCode":"\n    def __init__(self, quant_config: MarlinConfig):\n        self.quant_config = quant_config\n\n    def create_weights(\n        self,\n        layer: torch.nn.Module,\n        input_size_per_partition: int,\n        output_partition_sizes: list[int],\n        input_size: int,\n        output_size: int,\n        params_dtype: torch.dtype,\n        **extra_weight_attrs,\n    ):\n        del output_size  # Unused.\n        weight_loader = extra_weight_attrs[\"weight_loader\"]\n\n        if params_dtype != torch.float16:\n            raise ValueError(\n                f\"The params dtype must be float16, but got {params_dtype}\"\n            )\n\n        # Validate output_size_per_partition\n        output_size_per_partition = sum(output_partition_sizes)\n        if output_size_per_partition % self.quant_config.min_n_threads != 0:\n            raise ValueError(\n                f\"Weight output_size_per_partition = \"\n                f\"{output_size_per_partition} is not divisible by \"\n                f\"min_n_threads = {self.quant_config.min_n_threads}.\"\n            )\n        if output_size_per_partition % self.quant_config.pack_factor != 0:\n            raise ValueError(\n                f\"Weight output_size_per_partition = \"\n                f\"{output_size_per_partition} is not divisible by \"\n                f\"pack_factor = {self.quant_config.pack_factor}.\"\n            )\n","sourceCodeStart":737,"sourceCodeEnd":773,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/marlin_utils.py#L737-L773","documentation":"Marlin repacking in this code path is implemented for FP16 weights only: create_weights raises ValueError if params_dtype is not torch.float16 (BF16, FP32, FP8 all rejected), because the Marlin kernel and its weight-packing helpers assume half precision.","triggerScenarios":"Launching with --dtype bfloat16 or float32 on a model routed to this Marlin method; a model config specifying torch_dtype bfloat16 combined with a GPTQ/AWQ Marlin checkpoint.","commonSituations":"Modern models defaulting to BF16 being served with older Marlin-only quant checkpoints; forcing --dtype float32 for debugging quantized models.","solutions":["Launch with --dtype float16 (or remove the override so the FP16 default applies for this checkpoint)","Use a non-Marlin backend (--quantization gptq) if BF16 serving is mandatory","Confirm the checkpoint's torch_dtype in config.json is float16"],"exampleFix":"# before\npython -m sglang.launch_server --model gptq-model --dtype bfloat16\n# after\npython -m sglang.launch_server --model gptq-model --dtype float16","handlingStrategy":"validation","validationCode":"import torch\nassert server_args.dtype in (None, \"float16\", \"half\") or torch.dtype(server_args.dtype) is torch.float16, \"Marlin path requires FP16\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Serve Marlin checkpoints with --dtype float16 or no dtype override","Avoid --dtype bfloat16 with GPTQ/AWQ Marlin models"],"tags":["marlin","dtype","float16","config-validation"],"backgroundTag":"dtype-mismatch","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}