sgl-project/sglang · error · NotImplementedError

MIXED_PRECISION layer group {tail!r} uses unsupported quant

Error message

MIXED_PRECISION layer group {tail!r} uses unsupported quant algo {algo!r}; online requantization supports NVFP4 (-> MXFP4) and FP8 (kept as-is) only.

What it means

In the same MIXED_PRECISION builder, each per-tail group whose algo is consistent is then mapped to a target spec: NVFP4/W4A16_NVFP4 -> MXFP4 target, FP8 -> kept as FP8. Any other algo (e.g. INT8, INT4, W8A8-INT8) has no online requantization path in sglang, so NotImplementedError is raised naming the offending group and algo.

Source

Thrown at python/sglang/srt/layers/quantization/quark/quark.py:238

    layer_quant_config: Dict[str, Any] = {}
    has_nvfp4 = False
    for tail, algos in tail_algos.items():
        if len(algos) != 1:
            raise NotImplementedError(
                f"MIXED_PRECISION layer group {tail!r} has inconsistent "
                f"quant algos across layers: {sorted(algos)}. SGLang requires "
                "all layers in a group to share one algo."
            )
        algo = next(iter(algos))
        pattern = "*" + tail
        if algo in ("NVFP4", "W4A16_NVFP4"):
            layer_quant_config[pattern] = _MXFP4_TARGET_SPEC
            has_nvfp4 = True
        elif algo == "FP8":
            layer_quant_config[pattern] = fp8_spec
        else:
            raise NotImplementedError(
                f"MIXED_PRECISION layer group {tail!r} uses unsupported "
                f"quant algo {algo!r}; online requantization supports NVFP4 "
                "(-> MXFP4) and FP8 (kept as-is) only."
            )
    return layer_quant_config, has_nvfp4


def _build_excluded_fp8_config(config: Dict[str, Any]) -> Optional["Fp8Config"]:
    """Build a load-as-is `Fp8Config` for the excluded layers of a
    mixed-precision NVFP4 source, or None if excluded layers are bf16.

    Two producer conventions are handled:

    - FP8-serialized base (``quant_method == "fp8"``, e.g.
      DeepSeek-V4-Pro-NVFP4): the routed experts are NVFP4 (requantized to
      MXFP4) while attn / shared_experts stay FP8 and are listed in the
      excludes. Those FP8 layers load through `Fp8LinearMethod`;
      ``weight_block_size`` selects block (e.g. ``[128, 128]``) vs per-tensor

View on GitHub (pinned to 0132848349)

Solutions

1. Re-quantize or re-export the checkpoint so every layer group uses NVFP4 or FP8 only.
2. Serve the model with its native quark quantization (omit --quantization quark_mxfp4) so the unsupported algos are handled by their own quark paths (if supported).
3. Upgrade sglang — additional algos may have been added to the online requantization allowlist.

Example fix

# before
# MIXED_PRECISION config with '*.gate_up_proj' algo INT8 -> NotImplementedError

# after
# re-export with '*.gate_up_proj' as FP8 -> kept as-is, or NVFP4 -> requantized to MXFP4

SUPPORTED = {"NVFP4", "W4A16_NVFP4", "FP8"}
algos = {a for algos in tail_algos.values() for a in algos}
unsupported = algos - SUPPORTED
if unsupported:
    raise RuntimeError(f"Algos not supported for online requantization: {unsupported}")

def algo_supported_for_online_requant(algo: str) -> bool:
    return algo in {"NVFP4", "W4A16_NVFP4", "FP8"}

try:
    QuarkConfig.from_config(quant_config=config, hf_config=hf_config)
except NotImplementedError as e:
    if "unsupported quant algo" in str(e):
        load_with_native_quark_instead(model_path)
    raise

When it happens

Trigger: A MIXED_PRECISION quark checkpoint where some tail group uses an algo other than NVFP4, W4A16_NVFP4, or FP8, loaded with --quantization quark_mxfp4 (from_config -> _build_mixed_precision_layer_quant_config).

Common situations: Mixed checkpoints that combine INT8 or INT4 sublayers with FP4 sublayers; IBM Granite-style quark models with newer algo names; attempting to serve a partially-quantized checkpoint through the online MXFP4 requantization scheme.

Related errors

• MIXED_PRECISION layer group {tail!r} has inconsistent quant
• MIXED_PRECISION checkpoint has no NVFP4 layers to requantize
• Online MXFP4 requantization from compressed-tensors NVFP4 ch
• The package `amd-quark` is required to use MX-FP4 models. Pl
• Unsupported online_scheme: {online_scheme}

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/023bfea786a6e2a4. Report an issue: GitHub.