{"record":{"id":"b6ae921548445ee6","repo":"sgl-project/sglang","slug":"static-compressed-tensors-scheme-is-not-yet-suppor","errorCode":null,"errorMessage":"Static compressed-tensors scheme is not yet supported on NPU.","messagePattern":"Static compressed-tensors scheme is not yet supported on NPU\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py","lineNumber":191,"sourceCode":"    ) -> torch.Tensor:\n        # TODO: add cutlass_scaled_mm_azp support\n        x_q, x_scale = per_token_quant_int8(x)\n\n        return int8_scaled_mm(\n            x_q, layer.weight, x_scale, layer.weight_scale, out_dtype=x.dtype, bias=bias\n        )\n\n\nclass NPUCompressedTensorsW8A8Int8(CompressedTensorsW8A8Int8):\n\n    def __init__(\n        self, strategy: str, is_static_input_scheme: bool, input_symmetric: bool\n    ):\n        super().__init__(strategy, is_static_input_scheme, input_symmetric)\n        # TODO: Currently, NPU kernel for static quant requires quant_bias field,\n        # which can't be replicated in compressed-tensors.\n        if self.is_static_input_scheme:\n            raise NotImplementedError(\n                \"Static compressed-tensors scheme is not yet supported on NPU.\"\n            )\n        self.kernel = NPUW8A8Int8DynamicLinearMethod()\n\n    @classmethod\n    def get_min_capability(cls) -> int:\n        return NotImplementedError\n\n    def process_weights_after_loading(self, layer):\n        return self.kernel.process_weights_after_loading(layer)\n\n    def apply_weights(self, layer, x, bias):\n        return self.kernel.apply(layer, x, bias)\n","sourceCodeStart":173,"sourceCodeEnd":205,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py#L173-L205","documentation":"The NPU variant of the W8A8 INT8 scheme only implements dynamic input quantization; the NPU static-quant kernel needs a quant_bias field that compressed-tensors checkpoints don't carry, so static input schemes raise NotImplementedError at init.","triggerScenarios":"Loading a compressed-tensors INT8 model with static (calibrated) input scales on an NPU device, selecting CompressedTensorsW8A8Int8NPUMethod.","commonSituations":"Running calibrated INT8 checkpoints on Ascend NPU hardware.","solutions":["Re-quantize with dynamic input quantization (input_quant.dynamic = true)","Use a different backend (GPU) if static INT8 is required","Wait for/track upstream support for static compressed-tensors on NPU"],"exampleFix":"// before\n\"input_quant\": {\"dynamic\": false}\n// after\n\"input_quant\": {\"dynamic\": true}","handlingStrategy":"validation","validationCode":"if is_npu():\n    assert cfg[\"quantization_config\"][\"input_quant\"].get(\"dynamic\", True), \"NPU needs dynamic input quant\"","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Maintain separate dynamic-quant checkpoints for NPU targets"],"tags":["npu","int8","quantization","static-scales","not-implemented"],"backgroundTag":"platform-unsupported-quantization-mode","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}