{"record":{"id":"ce95a19d630f66ea","repo":"sgl-project/sglang","slug":"block-quantized-lm-head-is-not-supported-use-chan","errorCode":null,"errorMessage":"Block-quantized lm_head is not supported; use channel or tensor weight scales for the head.","messagePattern":"Block-quantized lm_head is not supported; use channel or tensor weight scales for the head\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py","lineNumber":1033,"sourceCode":"        # When several config groups name the head, the first target in config\n        # order wins — the same first-match rule find_matched_target applies\n        # to every other layer.\n        matched_target = next(\n            (\n                target\n                for target in self.target_scheme_map\n                if check_equal_or_regex_match(layer_name=layer_name, targets=[target])\n            ),\n            None,\n        )\n        if matched_target is None:\n            return None\n        weights = self.target_scheme_map[matched_target].get(\"weights\")\n        if weights is not None and weights.block_structure:\n            # The vocab-parallel weight loader shards output_dim=0 params by\n            # vocab index; a block weight_scale's first dim is vocab/block_n,\n            # which that loader cannot shard or even load at TP=1.\n            raise NotImplementedError(\n                \"Block-quantized lm_head is not supported; use channel or \"\n                \"tensor weight scales for the head.\"\n            )\n        return self.get_linear_scheme(\n            layer=layer, layer_name=layer_name, matched_target=matched_target\n        )\n\n    def get_scheme_dict(\n        self,\n        layer: torch.nn.Module,\n        layer_name: str | None = None,\n        matched_target: str | None = None,\n    ) -> dict[str, QuantizationArgs | str | None] | None:\n        \"\"\"\n        Extract the QuantizationArgs for a given layer.\n\n        A caller that already resolved the layer's target (e.g. via\n        suffix-aware matching) passes it as ``matched_target`` to skip","sourceCodeStart":1015,"sourceCodeEnd":1051,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py#L1015-L1051","documentation":"get_lm_head_scheme found that the matched lm_head target's weight scheme uses block_structure scales (block-quantized, e.g. 2x4 or 128x128 blocks). The vocab-parallel loader shards output_dim=0 params by vocab index and cannot shard/load a block weight_scale whose first dim is vocab/block_n, so this is rejected even at TP=1.","triggerScenarios":"A compressed-tensors checkpoint whose quantization config targets lm_head (it is not in the ignore list) with block-structured weight scales, e.g. INT8 block-quant recipe applied to all Linear layers including lm_head.","commonSituations":"Quantizing with llmcompressor recipe that doesn't ignore lm_head (ignore=[\"lm_head\"]) and uses a block modifier like Int8WeightOnlyModifier(group_size=...) — but block structured; DeepSeek-style block-quantized checkpoints accidentally covering lm_head.","solutions":["Re-quantize with lm_head excluded: ignore=[\"lm_head\"] in the llmcompressor recipe","Re-quantize lm_head with channel-wise or tensor (per-tensor) weight scales instead of block scales","If re-quantization is impossible, strip lm_head from the compressed-tensors config so it loads unquantized"],"exampleFix":"# before\nrecipe = [BlockQuantModifier(targets=\"Linear\")]  # covers lm_head\n# after\nrecipe = [BlockQuantModifier(targets=\"Linear\", ignore=[\"lm_head\"])]","handlingStrategy":"validation","validationCode":"cfg = model_cfg[\"quantization_config\"]\nfor target, scheme in cfg[\"config\"][\"targets_map\"].items() if \"targets_map\" in cfg[\"config\"] else []:\n    pass\n# simplest: check the lm_head target's weights scheme\nw = None\nfor t, m in (cfg[\"config\"].get(\"targets_map\") or {}).items():\n    if \"lm_head\" in t:\n        w = m.get(\"weights\")\nif w and w.get(\"block_structure\"):\n    raise SystemExit(\"re-quantize without block scales on lm_head\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Always ignore lm_head in llmcompressor recipes (ignore=[\"lm_head\"])","Never apply block-structured modifiers to output-embedding-tied layers","Validate the quantization_config covers only intended targets"],"tags":["quantization","lm-head","block-quantization","weight-loading","tp-sharding"],"backgroundTag":"unsupported-quantization-scheme","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}