sgl-project/sglang · error · ValueError

Found different quantization schemes for {shard_proj_names}

Error message

Found different quantization schemes for {shard_proj_names} in {layer_name}. SGLang requires all to use the same scheme.

What it means

When SGLang maps a fused layer (e.g. gate_proj+up_proj → gate_up_proj) onto compressed-tensors targets, all shards of a fused projection must agree on being quantized or not. If shard 0 is ignored (unquantized) but shard 1 is quantized (or vice versa), this ValueError is raised from should_ignore_layer.

Source

Thrown at python/sglang/srt/layers/quantization/compressed_tensors/utils.py:62

        shard_names = [
            layer_name.replace(proj_name, shard_proj_name)
            for shard_proj_name in shard_proj_names
        ]

        # Layer should be ignored if shards are ignored.
        should_ignore_layer = None
        for shard_name in shard_names:
            should_ignore_shard = check_equal_or_regex_match(
                layer_name=shard_name, targets=ignore
            )

            # If shard_idx=0, set layer ignore to match shard.
            if should_ignore_layer is None:
                should_ignore_layer = should_ignore_shard

            # If shard_idx=1+ confirm scheme matches prior shards.
            elif should_ignore_shard != should_ignore_layer:
                raise ValueError(
                    f"Found different quantization schemes for "
                    f"{shard_proj_names} in {layer_name}. SGLang "
                    "requires all to use the same scheme."
                )

    # Unfused layers like down_proj and o_proj will match
    # the safetensors checkpoint already.
    else:
        should_ignore_layer = check_equal_or_regex_match(
            layer_name=layer_name, targets=ignore
        )

    assert should_ignore_layer is not None
    return should_ignore_layer


def check_equal_or_regex_match(layer_name: str, targets: Iterable[str]) -> bool:
    """

View on GitHub (pinned to 0132848349)

Solutions

Re-quantize so all shards of each fused projection share the same scheme (quantize all of q/k/v or none; gate+up together)
If partial quant is intended, disable fusion of those projections or edit targets so they align
Inspect the compressed-tensors config_targets list for missing shard entries

Example fix

// before: targets cover model.layers.0.self_attn.q_proj only
// after: include k_proj and v_proj (or remove q_proj) so the fused projection is uniform

Defensive patterns

Strategy: validation

Validate before calling

targets = set(cfg["quantization_config"].get("config_lists") or [])
# ensure fused projection shards are uniformly covered
for fused in ("qkv_proj", "gate_up_proj"):
    shards = {"q_proj","k_proj","v_proj"} if fused=="qkv_proj" else {"gate_proj","up_proj"}
    covered = [any(s in t for t in targets) for s in shards]
    assert all(covered) or not any(covered), f"mixed quantization across {fused} shards"

Prevention

Quantize projections in fused groups (all of q/k/v, gate+up together)
Use wildcard targets covering whole module groups

When it happens

Trigger: A checkpoint where q_proj is quantized but k_proj/v_proj are not (or gate_proj without up_proj) — the fused layer's shards disagree on should_ignore.

Common situations: Partial-quantization recipes that quantize attention but not KV projections, then fuse; mixed-precision models from custom finetunes.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/8d5a1759d3915185. Report an issue: GitHub.