sgl-project/sglang · error · NotImplementedError

MIXED_PRECISION layer group {tail!r} has inconsistent quant

Error message

MIXED_PRECISION layer group {tail!r} has inconsistent quant algos across layers: {sorted(algos)}. SGLang requires all layers in a group to share one algo.

What it means

For MIXED_PRECISION quark checkpoints, _build_mixed_precision_layer_quant_config groups layers by name tail (e.g. 'proj', 'mlp') and requires each group to use exactly one quant algo. If layers sharing a suffix were quantized with different algos, the mapping from group -> target spec is ambiguous, so sglang raises NotImplementedError.

Source

Thrown at python/sglang/srt/layers/quantization/quark/quark.py:225

    """Collapse a per-layer {name: quant_algo} map into a compact
    `layer_quant_config` keyed by fnmatch glob patterns.
    """
    # suffix tail -> set of algos seen (to detect inconsistency)
    tail_algos: Dict[str, set] = {}
    for name, algo in layer_map.items():
        # Suffix after the last `.layers.<idx>.` (or the whole name if
        # unindexed); this is the part shared across all layer indices.
        tail = re.split(r"\.layers\.\d+\.", name, maxsplit=1)[-1]
        tail_algos.setdefault(tail, set()).add(algo)

    fp8_is_dynamic = _fp8_is_dynamic_from_config_groups(config_groups or {})
    fp8_spec = _fp8_per_tensor_spec(is_dynamic_input=fp8_is_dynamic)

    layer_quant_config: Dict[str, Any] = {}
    has_nvfp4 = False
    for tail, algos in tail_algos.items():
        if len(algos) != 1:
            raise NotImplementedError(
                f"MIXED_PRECISION layer group {tail!r} has inconsistent "
                f"quant algos across layers: {sorted(algos)}. SGLang requires "
                "all layers in a group to share one algo."
            )
        algo = next(iter(algos))
        pattern = "*" + tail
        if algo in ("NVFP4", "W4A16_NVFP4"):
            layer_quant_config[pattern] = _MXFP4_TARGET_SPEC
            has_nvfp4 = True
        elif algo == "FP8":
            layer_quant_config[pattern] = fp8_spec
        else:
            raise NotImplementedError(
                f"MIXED_PRECISION layer group {tail!r} uses unsupported "
                f"quant algo {algo!r}; online requantization supports NVFP4 "
                "(-> MXFP4) and FP8 (kept as-is) only."
            )
    return layer_quant_config, has_nvfp4

View on GitHub (pinned to 0132848349)

Solutions

1. Inspect the checkpoint's quant config config_groups and list each layer's quant algo; find layers sharing a tail with differing algos.
2. Re-export the checkpoint so all layers within each tail group share one algo (NVFP4 or FP8), or rely on the native quark method instead of --quantization quark_mxfp4.
3. File/support a feature request in sglang for finer-grained mixed-precision patterns if the layout is intentional.

Example fix

# before
# config_groups: self_attn.q_proj -> NVFP4, mlp.gate_proj -> FP8 (same tail 'proj' -> inconsistent)

# after
# all '*proj' layers share one algo, e.g. all NVFP4 -> pattern "*proj" maps to MXFP4 target spec

algos_by_tail = collect_tail_algos(config["config_groups"])  # tail -> set of algos
bad = {t: sorted(a) for t, a in algos_by_tail.items() if len(a) != 1}
if bad:
    raise RuntimeError(f"Inconsistent algos per tail group: {bad}")

def is_consistent_mixed_precision(config: dict) -> bool:
    groups = config.get("config_groups", {})
    tails = {}
    for g in groups.values():
        tail = g["weights"].get("symmetric") and g  # placeholder: derive tail from layer names
    return all(len(v) == 1 for v in tails.values()) if tails else True

try:
    QuarkConfig.from_config(quant_config=config, hf_config=hf_config)
except NotImplementedError as e:
    if "inconsistent quant algos" in str(e):
        fail_ci_with_checkpoint_reexport_hint(e)
    raise

When it happens

Trigger: Loading a MIXED_PRECISION quark checkpoint with --quantization quark_mxfp4 where, inside config_groups, two layers matching the same tail pattern (e.g. self_attn.q_proj and mlp.gate_proj if grouped by 'proj') report different quant algos like NVFP4 vs FP8 vs INT8.

Common situations: Hand-edited or custom-produced mixed-precision quark configs that quantize different sublayers differently but with colliding name tails; checkpoints exported by newer tooling with finer-grained mixed schemes than sglang's online requantization supports.

Related errors

• MIXED_PRECISION layer group {tail!r} uses unsupported quant
• MIXED_PRECISION checkpoint has no NVFP4 layers to requantize
• Online MXFP4 requantization from compressed-tensors NVFP4 ch
• Unsupported online_scheme: {online_scheme}
• The package `amd-quark` is required to use MX-FP4 models. Pl

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/3639bbb763315cdf. Report an issue: GitHub.