{"record":{"id":"b10ca40d7fb46045","repo":"sgl-project/sglang","slug":"detected-some-but-not-all-shards-of-prefix-are-q","errorCode":null,"errorMessage":"Detected some but not all shards of {prefix} are quantized. All shards of fused layers to have the same precision.","messagePattern":"Detected some but not all shards of (.+?) are quantized\\. All shards of fused layers to have the same precision\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/quantization/modelslim/modelslim.py","lineNumber":459,"sourceCode":"        # adapted from vllm.model_executor.layers.quantization.utils.quant_utils.is_layer_skipped\n        proj_name = prefix.split(\".\")[-1]\n        if proj_name in fused_mapping:\n            shard_prefixes = [\n                prefix.replace(proj_name, shard_proj_name)\n                for shard_proj_name in fused_mapping[proj_name]\n            ]\n\n            is_skipped = None\n            for shard_prefix in shard_prefixes:\n                shard_prefix = self._resolve_quant_prefix(shard_prefix)\n                is_shard_skipped = (\n                    self.quant_description.get(shard_prefix + \".weight\", \"\") == \"FLOAT\"\n                )\n\n                if is_skipped is None:\n                    is_skipped = is_shard_skipped\n                elif is_shard_skipped != is_skipped:\n                    raise ValueError(\n                        f\"Detected some but not all shards of {prefix} \"\n                        \"are quantized. All shards of fused layers \"\n                        \"to have the same precision.\"\n                    )\n        else:\n            prefix = self._resolve_quant_prefix(prefix)\n            is_skipped = self.quant_description.get(prefix + \".weight\", \"\") == \"FLOAT\"\n\n        assert is_skipped is not None\n        return is_skipped\n\n    def get_scaled_act_names(self) -> List[str]:\n        return []\n\n\nclass ModelSlimLinearMethod(_NPULinearMethodBase):\n\n    def __init__(self, quantization_config: ModelSlimConfig):","sourceCodeStart":441,"sourceCodeEnd":477,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/modelslim/modelslim.py#L441-L477","documentation":"is_layer_skipped detects that a fused layer's shards (e.g. gate/up/down of a fused QKV or MoE projection, matched by shard_prefix + '.weight') disagree: some shards are marked 'FLOAT' (unquantized) in quant_description while others are quantized. Fused kernels cannot mix precisions across shards, so this is rejected.","triggerScenarios":"Calling the skip-check path with a fused-layer prefix where, across the shard prefixes matched, quant_description[shard + '.weight'] == 'FLOAT' for some shards but a quantized scheme for others.","commonSituations":"msModelSlim run with mixed precision rules quantizing q/k but not v (or gate but not up); partial re-quantization of a checkpoint; manually editing the description to skip one projection of a fused layer.","solutions":["Open the quant_description and make all shards of the fused layer the same precision (all quantized or all 'FLOAT')","Re-run msModelSlim with consistent include/exclude rules at the fused-layer granularity, never per projection","If a shard must stay unquantized, split the fused op in the model config or quantize all shards","Verify the shard_prefix matching isn't over-matching unrelated layers with different precisions"],"exampleFix":"// before (description JSON)\n\"...self_attn.q_proj.weight\": \"W8A8\",\n\"...self_attn.k_proj.weight\": \"W8A8\",\n\"...self_attn.v_proj.weight\": \"FLOAT\"\n// after\n\"...self_attn.q_proj.weight\": \"W8A8\",\n\"...self_attn.k_proj.weight\": \"W8A8\",\n\"...self_attn.v_proj.weight\": \"W8A8\"","handlingStrategy":"validation","validationCode":"import re\n\ndef fused_shards_consistent(qd: dict, prefixes: list[str]) -> bool:\n    flags = [qd.get(p + \".weight\", \"\") == \"FLOAT\" for p in prefixes]\n    return all(f == flags[0] for f in flags)\n\n# e.g. for fused qkv:\nassert fused_shards_consistent(config.quant_description,\n    [f\"{blk}.self_attn.{n}_proj\" for n in (\"q\", \"k\", \"v\")])","typeGuard":null,"tryCatchPattern":"try:\n    config.get_quant_method(layer, prefix)\nexcept ValueError as e:\n    if \"some but not all shards\" in str(e):\n        # equalize precisions in quant_description, then retry load\n        raise","preventionTips":["Apply quantization include/exclude rules at fused-layer granularity","Automated check: for each fused layer, assert all shard entries share FLOAT vs quantized status","Re-quantize instead of hand-editing when precisions disagree"],"tags":["modelslim","quantization","fused-layers","mixed-precision","config"],"backgroundTag":"mixed-precision-fused-layer","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}