sgl-project/sglang · error · ValueError

Fused MoE layer '{layer_name}' requires consistent quant con

Error message

Fused MoE layer '{layer_name}' requires consistent quant config for all sub-layers

What it means

AutoRoundConfig.get_layer_config detects that the queried layer is a fused MoE layer whose sub-layers (experts) have differing per-layer quant configs in extra_config. A fused MoE kernel needs one uniform config, so inconsistency raises.

Source

Thrown at python/sglang/srt/layers/quantization/auto_round.py:255

        # 2. Determine whether layer should be quantized
        quantized = not isinstance(layer, ParallelLMHead)
        if self.block_name_to_quantize:
            quantized = any(
                layer_name.startswith(name) for name in self.block_name_to_quantize
            )

        # 3. Handle fused MoE
        if self.extra_config and "fusedmoe" in layer.__class__.__name__.lower():
            moe_configs = [
                get_config(name, quantized)
                for name in self.extra_config
                if name.startswith(layer_name)
            ]
            if moe_configs:
                if len(set(moe_configs)) == 1:
                    return moe_configs[0]
                raise ValueError(
                    f"Fused MoE layer '{layer_name}' requires "
                    f"consistent quant config for all sub-layers"
                )

        # 4. Handle fused QKV or other patterns
        if self.extra_config:
            for fusion_key, sub_keys in self.packed_modules_mapping.items():
                if fusion_key in layer_name and layer_name.count(fusion_key) == 1:
                    sub_names = [
                        layer_name.replace(fusion_key, sub_key) for sub_key in sub_keys
                    ]
                    sub_configs = [get_config(name, quantized) for name in sub_names]
                    if len(set(sub_configs)) == 1:
                        return sub_configs[0]
                    raise ValueError(
                        f"Fused module '{layer_name}' requires "
                        f"consistent quant config for {sub_names}"
                    )

View on GitHub (pinned to 0132848349)

Solutions

Make the extra_config entries identical for all sub-layers of the fused MoE layer (remove per-expert overrides)
Re-export the model with uniform quantization for the MoE block
If mixed-precision experts are required, use a path that supports per-expert methods instead of fused MoE

Example fix

// before (extra_config)
{"model.layers.0.mlp.experts.0": {...4bit...}, "model.layers.0.mlp.experts.1": {...8bit...}}
// after
{"model.layers.0.mlp": {...4bit uniform...}}

Defensive patterns

Strategy: validation

Validate before calling

import collections
groups = collections.defaultdict(set)
for name, c in extra_config.items():
    prefix = name.rsplit(".experts.", 1)[0]
    groups[prefix].add(json.dumps(c, sort_keys=True))
assert all(len(v) == 1 for v in groups.values()), "MoE sub-layer configs differ"

Try / catch

try:
    cfg.get_layer_config(layer_name)
except ValueError as e:
    if "Fused MoE layer" in str(e): unify_expert_configs_and_retry()
    raise

Prevention

Never write per-expert quant overrides for fused MoE
Validate extra_config uniformity in a pre-launch check script

When it happens

Trigger: An AutoRound checkpoint whose quant_config.json / extra_config specifies different quant settings for different experts under the same MoE layer prefix (e.g. some experts 4-bit, some 8-bit), then calling get_quant_method on that layer.

Common situations: Mixed-precision expert quantization exports, hand-edited extra_config, or partial per-layer overrides that unintentionally split across expert indices.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/16e51bb9e952f5f1. Report an issue: GitHub.