sgl-project/sglang · error · NotImplementedError

Block-quantized lm_head is not supported; use channel or ten

Error message

Block-quantized lm_head is not supported; use channel or tensor weight scales for the head.

What it means

get_lm_head_scheme found that the matched lm_head target's weight scheme uses block_structure scales (block-quantized, e.g. 2x4 or 128x128 blocks). The vocab-parallel loader shards output_dim=0 params by vocab index and cannot shard/load a block weight_scale whose first dim is vocab/block_n, so this is rejected even at TP=1.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors.py:1033

        # When several config groups name the head, the first target in config
        # order wins — the same first-match rule find_matched_target applies
        # to every other layer.
        matched_target = next(
            (
                target
                for target in self.target_scheme_map
                if check_equal_or_regex_match(layer_name=layer_name, targets=[target])
            ),
            None,
        )
        if matched_target is None:
            return None
        weights = self.target_scheme_map[matched_target].get("weights")
        if weights is not None and weights.block_structure:
            # The vocab-parallel weight loader shards output_dim=0 params by
            # vocab index; a block weight_scale's first dim is vocab/block_n,
            # which that loader cannot shard or even load at TP=1.
            raise NotImplementedError(
                "Block-quantized lm_head is not supported; use channel or "
                "tensor weight scales for the head."
            )
        return self.get_linear_scheme(
            layer=layer, layer_name=layer_name, matched_target=matched_target
        )

    def get_scheme_dict(
        self,
        layer: torch.nn.Module,
        layer_name: str | None = None,
        matched_target: str | None = None,
    ) -> dict[str, QuantizationArgs | str | None] | None:
        """
        Extract the QuantizationArgs for a given layer.

        A caller that already resolved the layer's target (e.g. via
        suffix-aware matching) passes it as ``matched_target`` to skip

View on GitHub (pinned to 0132848349)

Solutions

Re-quantize with lm_head excluded: ignore=["lm_head"] in the llmcompressor recipe
Re-quantize lm_head with channel-wise or tensor (per-tensor) weight scales instead of block scales
If re-quantization is impossible, strip lm_head from the compressed-tensors config so it loads unquantized

Example fix

# before
recipe = [BlockQuantModifier(targets="Linear")]  # covers lm_head
# after
recipe = [BlockQuantModifier(targets="Linear", ignore=["lm_head"])]

Defensive patterns

Strategy: validation

Validate before calling

cfg = model_cfg["quantization_config"]
for target, scheme in cfg["config"]["targets_map"].items() if "targets_map" in cfg["config"] else []:
    pass
# simplest: check the lm_head target's weights scheme
w = None
for t, m in (cfg["config"].get("targets_map") or {}).items():
    if "lm_head" in t:
        w = m.get("weights")
if w and w.get("block_structure"):
    raise SystemExit("re-quantize without block scales on lm_head")

Prevention

Always ignore lm_head in llmcompressor recipes (ignore=["lm_head"])
Never apply block-structured modifiers to output-embedding-tied layers
Validate the quantization_config covers only intended targets

When it happens

Trigger: A compressed-tensors checkpoint whose quantization config targets lm_head (it is not in the ignore list) with block-structured weight scales, e.g. INT8 block-quant recipe applied to all Linear layers including lm_head.

Common situations: Quantizing with llmcompressor recipe that doesn't ignore lm_head (ignore=["lm_head"]) and uses a block modifier like Int8WeightOnlyModifier(group_size=...) — but block structured; DeepSeek-style block-quantized checkpoints accidentally covering lm_head.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/ce95a19d630f66ea. Report an issue: GitHub.