sgl-project/sglang · error · ValueError

Unsupported params_dtype: {params_dtype}

Error message

Unsupported params_dtype: {params_dtype}

What it means

Raised while creating quantized weights for the ModelSlim W8A8 INT8 scheme. The dequantization scale tensor's dtype is derived from the layer's params_dtype: bfloat16 maps to float32 scales and float16 maps to int64 scales (NPU kernel requirement). Any other dtype — typically float32 — has no valid deq_scale representation and is rejected.

Source

Thrown at python/sglang/srt/layers/quantization/modelslim/schemes/modelslim_w8a8_int8.py:101

                data=torch.empty(1, dtype=params_dtype),
                weight_loader=weight_loader,
            )
            input_offset.ignore_warning = True
            layer.register_parameter("input_offset", input_offset)

            quant_bias = ChannelQuantScaleParameter(
                data=torch.empty(output_size_per_partition, dtype=torch.int32),
                output_dim=0,
                weight_loader=weight_loader,
            )
            layer.register_parameter("quant_bias", quant_bias)

            if params_dtype == torch.bfloat16:
                deq_scale_dtype = torch.float32
            elif params_dtype == torch.float16:
                deq_scale_dtype = torch.int64
            else:
                raise ValueError(f"Unsupported params_dtype: {params_dtype}")
            deq_scale = ChannelQuantScaleParameter(
                data=torch.empty(output_size_per_partition, dtype=deq_scale_dtype),
                output_dim=0,
                weight_loader=weight_loader,
            )
            layer.register_parameter("deq_scale", deq_scale)

    def process_weights_after_loading(self, layer: torch.nn.Module):
        self.kernel.process_weights_after_loading(layer)

    def apply_weights(
        self,
        layer: torch.nn.Module,
        x: torch.Tensor,
        bias: Optional[torch.Tensor] = None,
    ) -> torch.Tensor:
        return self.kernel.apply(layer, x, bias)

View on GitHub (pinned to 0132848349)

Solutions

1. Run with --dtype bfloat16 (or float16), matching the dtype the model was quantized/calibrated with
2. If float32 truly must be supported, patch create_weights to define a deq_scale dtype for it and verify the NPU kernel accepts it
3. Check the model config's torch_dtype field is bf16/fp16, not float32

Example fix

# before
python -m sglang.launch_server --model ... --dtype float32
# after
python -m sglang.launch_server --model ... --dtype bfloat16

import torch
assert params_dtype in (torch.bfloat16, torch.float16), (
    f"params_dtype {params_dtype} unsupported; use bf16/fp16"
)
quant_method.create_weights(layer, params_dtype, ...)

def is_supported_dtype(d: torch.dtype) -> TypeGuard[torch.dtype]:
    return d in (torch.bfloat16, torch.float16)

When it happens

Trigger: Calling create_weights on the W8A8 INT8 quant method with layer.params_dtype set to torch.float32 (or any dtype other than torch.bfloat16 / torch.float16), e.g. by running the server with --dtype float32 on a ModelSlim-quantized NPU model.

Common situations: User forces --dtype float32 for debugging or CPU-side comparison; or a checkpoint/config carries float32 as the default dtype and it propagates into the quantized layer creation.

Related errors

• Unsupported ascend_dispatcher_output_dtype: {self.ascend_dis
• weight_prefix must be 'w13' or 'w2', got '{weight_prefix}'
• weight_prefix must be 'w13' or 'w2', got '{weight_prefix}'
• {name} must be float32, got {scale.dtype}
• NPU packed attention requires q, k, and v with the same dtyp

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/81ecec96c92ac141. Report an issue: GitHub.