sgl-project/sglang · error · NotImplementedError

Static compressed-tensors scheme is not yet supported on NPU

Error message

Static compressed-tensors scheme is not yet supported on NPU.

What it means

The NPU variant of the W8A8 INT8 scheme only implements dynamic input quantization; the NPU static-quant kernel needs a quant_bias field that compressed-tensors checkpoints don't carry, so static input schemes raise NotImplementedError at init.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py:191

    ) -> torch.Tensor:
        # TODO: add cutlass_scaled_mm_azp support
        x_q, x_scale = per_token_quant_int8(x)

        return int8_scaled_mm(
            x_q, layer.weight, x_scale, layer.weight_scale, out_dtype=x.dtype, bias=bias
        )


class NPUCompressedTensorsW8A8Int8(CompressedTensorsW8A8Int8):

    def __init__(
        self, strategy: str, is_static_input_scheme: bool, input_symmetric: bool
    ):
        super().__init__(strategy, is_static_input_scheme, input_symmetric)
        # TODO: Currently, NPU kernel for static quant requires quant_bias field,
        # which can't be replicated in compressed-tensors.
        if self.is_static_input_scheme:
            raise NotImplementedError(
                "Static compressed-tensors scheme is not yet supported on NPU."
            )
        self.kernel = NPUW8A8Int8DynamicLinearMethod()

    @classmethod
    def get_min_capability(cls) -> int:
        return NotImplementedError

    def process_weights_after_loading(self, layer):
        return self.kernel.process_weights_after_loading(layer)

    def apply_weights(self, layer, x, bias):
        return self.kernel.apply(layer, x, bias)

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize with dynamic input quantization (input_quant.dynamic = true)
2. Use a different backend (GPU) if static INT8 is required
3. Wait for/track upstream support for static compressed-tensors on NPU

Example fix

// before
"input_quant": {"dynamic": false}
// after
"input_quant": {"dynamic": true}

if is_npu():
    assert cfg["quantization_config"]["input_quant"].get("dynamic", True), "NPU needs dynamic input quant"

When it happens

Trigger: Loading a compressed-tensors INT8 model with static (calibrated) input scales on an NPU device, selecting CompressedTensorsW8A8Int8NPUMethod.

Common situations: Running calibrated INT8 checkpoints on Ascend NPU hardware.

Related errors

• Comfy INT8 embedding weights support lookup only
• The W8A8Int8 Fused MoE scheme is implemented only for NPU fo
• For INT8 Fused MoE layers, we require channelwise, dynamic p
• NPU packed attention does not support a sequence that is emp
• QVGPackedCausalKVCache does not support pinned-sink (longliv

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b6ae921548445ee6. Report an issue: GitHub.