sgl-project/sglang · error · ValueError

Fused module '{layer_name}' requires consistent quant config

Error message

Fused module '{layer_name}' requires consistent quant config for {sub_names}

What it means

For fused modules like QKV projections, get_layer_config resolves each sub-layer's config (by replacing the fusion key) and requires them all to match. If e.g. q/k/v have different extra_config entries, it raises rather than guessing one config.

Source

Thrown at python/sglang/srt/layers/quantization/auto_round.py:270

            if moe_configs:
                if len(set(moe_configs)) == 1:
                    return moe_configs[0]
                raise ValueError(
                    f"Fused MoE layer '{layer_name}' requires "
                    f"consistent quant config for all sub-layers"
                )

        # 4. Handle fused QKV or other patterns
        if self.extra_config:
            for fusion_key, sub_keys in self.packed_modules_mapping.items():
                if fusion_key in layer_name and layer_name.count(fusion_key) == 1:
                    sub_names = [
                        layer_name.replace(fusion_key, sub_key) for sub_key in sub_keys
                    ]
                    sub_configs = [get_config(name, quantized) for name in sub_names]
                    if len(set(sub_configs)) == 1:
                        return sub_configs[0]
                    raise ValueError(
                        f"Fused module '{layer_name}' requires "
                        f"consistent quant config for {sub_names}"
                    )

        # 5. Fallback or try a regular expression match
        return get_config(layer_name, quantized)

    def check_quantized(self, weight_bits: int) -> bool:
        return weight_bits < 16

    def check_cpu_support(self, weight_bits: int) -> None:
        if weight_bits != 4:
            raise ValueError(
                "SGLang's AutoRound CPU inference path currently supports "
                "only 4-bit AWQ/GPTQ checkpoints because it uses the Intel "
                f"AMX INT4 backend, but got {weight_bits}-bit."
            )
        if not _is_cpu_amx_available:

View on GitHub (pinned to 0132848349)

Solutions

Unify the extra_config entries for all sub-layers of the fused module
Delete the per-sub-layer overrides so the fused layer falls back to the default config
Re-run AutoRound calibration with symmetric per-module settings

Example fix

// before
{"...q_proj": {"bits": 4}, "...k_proj": {"bits": 8}}
// after
{"...self_attn": {"bits": 4}}

Defensive patterns

Strategy: validation

Validate before calling

FUSIONS = [("qkv_proj", ["q_proj", "k_proj", "v_proj"])]
for fused, subs in FUSIONS:
    vals = {json.dumps(extra_config.get(s), sort_keys=True) for s in subs if s in extra_config}
    assert len(vals) <= 1, f"inconsistent config for {fused}"

Try / catch

try:
    cfg.get_layer_config(layer_name)
except ValueError as e:
    if "Fused module" in str(e): unify_sublayer_configs()
    raise

Prevention

Keep calibration symmetric across q/k/v
Lint extra_config for sub-layer divergence before serving

When it happens

Trigger: An AutoRound extra_config where the sub-layers of a fused module differ — e.g. different entries for ...self_attn.q_proj vs ...k_proj — while the loader asks for the fused qkv_proj layer's config.

Common situations: Per-linear tuning scripts (AutoRound/autoscale) writing asymmetric configs, hand-edits, or checkpoints tuned with different bits for k/v projections.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/5973b3c51f39940c. Report an issue: GitHub.