sgl-project/sglang · error · ValueError

Detected some but not all shards of {prefix} are quantized.

Error message

Detected some but not all shards of {prefix} are quantized. All shards of fused layers to have the same precision.

What it means

is_layer_skipped detects that a fused layer's shards (e.g. gate/up/down of a fused QKV or MoE projection, matched by shard_prefix + '.weight') disagree: some shards are marked 'FLOAT' (unquantized) in quant_description while others are quantized. Fused kernels cannot mix precisions across shards, so this is rejected.

Source

Thrown at python/sglang/srt/layers/quantization/modelslim/modelslim.py:459

        # adapted from vllm.model_executor.layers.quantization.utils.quant_utils.is_layer_skipped
        proj_name = prefix.split(".")[-1]
        if proj_name in fused_mapping:
            shard_prefixes = [
                prefix.replace(proj_name, shard_proj_name)
                for shard_proj_name in fused_mapping[proj_name]
            ]

            is_skipped = None
            for shard_prefix in shard_prefixes:
                shard_prefix = self._resolve_quant_prefix(shard_prefix)
                is_shard_skipped = (
                    self.quant_description.get(shard_prefix + ".weight", "") == "FLOAT"
                )

                if is_skipped is None:
                    is_skipped = is_shard_skipped
                elif is_shard_skipped != is_skipped:
                    raise ValueError(
                        f"Detected some but not all shards of {prefix} "
                        "are quantized. All shards of fused layers "
                        "to have the same precision."
                    )
        else:
            prefix = self._resolve_quant_prefix(prefix)
            is_skipped = self.quant_description.get(prefix + ".weight", "") == "FLOAT"

        assert is_skipped is not None
        return is_skipped

    def get_scaled_act_names(self) -> List[str]:
        return []


class ModelSlimLinearMethod(_NPULinearMethodBase):

    def __init__(self, quantization_config: ModelSlimConfig):

View on GitHub (pinned to 0132848349)

Solutions

1. Open the quant_description and make all shards of the fused layer the same precision (all quantized or all 'FLOAT')
2. Re-run msModelSlim with consistent include/exclude rules at the fused-layer granularity, never per projection
3. If a shard must stay unquantized, split the fused op in the model config or quantize all shards
4. Verify the shard_prefix matching isn't over-matching unrelated layers with different precisions

Example fix

// before (description JSON)
"...self_attn.q_proj.weight": "W8A8",
"...self_attn.k_proj.weight": "W8A8",
"...self_attn.v_proj.weight": "FLOAT"
// after
"...self_attn.q_proj.weight": "W8A8",
"...self_attn.k_proj.weight": "W8A8",
"...self_attn.v_proj.weight": "W8A8"

import re

def fused_shards_consistent(qd: dict, prefixes: list[str]) -> bool:
    flags = [qd.get(p + ".weight", "") == "FLOAT" for p in prefixes]
    return all(f == flags[0] for f in flags)

# e.g. for fused qkv:
assert fused_shards_consistent(config.quant_description,
    [f"{blk}.self_attn.{n}_proj" for n in ("q", "k", "v")])

try:
    config.get_quant_method(layer, prefix)
except ValueError as e:
    if "some but not all shards" in str(e):
        # equalize precisions in quant_description, then retry load
        raise

When it happens

Trigger: Calling the skip-check path with a fused-layer prefix where, across the shard prefixes matched, quant_description[shard + '.weight'] == 'FLOAT' for some shards but a quantized scheme for others.

Common situations: msModelSlim run with mixed precision rules quantizing q/k but not v (or gate but not up); partial re-quantization of a checkpoint; manually editing the description to skip one projection of a fused layer.

Related errors

• No ModelSlim MoE scheme found for layer {prefix}
• Missing ModelSlim MoE quantization description for layer {pr
• The quantization method `{quantization}` is already exists.
• The quantization config must be a subclass of `QuantizationC
• Invalid quantization method: {quantization}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b10ca40d7fb46045. Report an issue: GitHub.