sgl-project/sglang · error · ValueError

The params dtype must be float16, but got {params_dtype}

Error message

The params dtype must be float16, but got {params_dtype}

What it means

Marlin repacking in this code path is implemented for FP16 weights only: create_weights raises ValueError if params_dtype is not torch.float16 (BF16, FP32, FP8 all rejected), because the Marlin kernel and its weight-packing helpers assume half precision.

Source

Thrown at python/sglang/srt/layers/quantization/marlin_utils.py:755

    def __init__(self, quant_config: MarlinConfig):
        self.quant_config = quant_config

    def create_weights(
        self,
        layer: torch.nn.Module,
        input_size_per_partition: int,
        output_partition_sizes: list[int],
        input_size: int,
        output_size: int,
        params_dtype: torch.dtype,
        **extra_weight_attrs,
    ):
        del output_size  # Unused.
        weight_loader = extra_weight_attrs["weight_loader"]

        if params_dtype != torch.float16:
            raise ValueError(
                f"The params dtype must be float16, but got {params_dtype}"
            )

        # Validate output_size_per_partition
        output_size_per_partition = sum(output_partition_sizes)
        if output_size_per_partition % self.quant_config.min_n_threads != 0:
            raise ValueError(
                f"Weight output_size_per_partition = "
                f"{output_size_per_partition} is not divisible by "
                f"min_n_threads = {self.quant_config.min_n_threads}."
            )
        if output_size_per_partition % self.quant_config.pack_factor != 0:
            raise ValueError(
                f"Weight output_size_per_partition = "
                f"{output_size_per_partition} is not divisible by "
                f"pack_factor = {self.quant_config.pack_factor}."
            )

View on GitHub (pinned to 0132848349)

Solutions

1. Launch with --dtype float16 (or remove the override so the FP16 default applies for this checkpoint)
2. Use a non-Marlin backend (--quantization gptq) if BF16 serving is mandatory
3. Confirm the checkpoint's torch_dtype in config.json is float16

Example fix

# before
python -m sglang.launch_server --model gptq-model --dtype bfloat16
# after
python -m sglang.launch_server --model gptq-model --dtype float16

import torch
assert server_args.dtype in (None, "float16", "half") or torch.dtype(server_args.dtype) is torch.float16, "Marlin path requires FP16"

When it happens

Trigger: Launching with --dtype bfloat16 or float32 on a model routed to this Marlin method; a model config specifying torch_dtype bfloat16 combined with a GPTQ/AWQ Marlin checkpoint.

Common situations: Modern models defaulting to BF16 being served with older Marlin-only quant checkpoints; forcing --dtype float32 for debugging quantized models.

Related errors

• Unknown dtype: {dtype}
• Humming does not support DeepEP {output_dtype} dispatch; use
• Currently, only group size 128 and -1 (channelwise) is suppo
• Unsupported content type ${header.content_type}
• Generate subcommand is not yet supported for model: {model_p

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b48cc5903a0fcf58. Report an issue: GitHub.