sgl-project/sglang · error · NotImplementedError

MIXED_PRECISION checkpoint has no NVFP4 layers to requantize

Error message

MIXED_PRECISION checkpoint has no NVFP4 layers to requantize; load it with its native quantization method instead of --quantization quark_mxfp4.

What it means

When a MIXED_PRECISION quark checkpoint is loaded with the quark_mxfp4 online requantization scheme, from_config requires at least one NVFP4 layer group to requantize into MXFP4. If the layer map contains only FP8 (or otherwise non-NVFP4) groups, has_nvfp4 is False and NotImplementedError tells you to use the checkpoint's native quantization instead.

Source

Thrown at python/sglang/srt/layers/quantization/quark/quark.py:436

    @classmethod
    def from_config(cls, config: dict[str, Any]) -> "QuarkConfig":
        # Requantization dispatch is gated on requantization_method, NOT on
        # quant_method. Quark-exported NVFP4 carries quant_method="quark" too
        if config.get("requantization_method") == "quark_mxfp4":
            hf_config = config["hf_config"]

            # Mixed-precision source: only the NVFP4 layers are requantized to
            # MXFP4; layers in other precisions (e.g. FP8) load through their
            # own scheme
            layer_map = _mixed_precision_layer_map(config)
            if layer_map is not None:
                config_groups = config.get("config_groups")
                layer_quant_config, has_nvfp4 = (
                    _build_mixed_precision_layer_quant_config(layer_map, config_groups)
                )
                if not has_nvfp4:
                    raise NotImplementedError(
                        "MIXED_PRECISION checkpoint has no NVFP4 layers to "
                        "requantize; load it with its native quantization "
                        "method instead of --quantization quark_mxfp4."
                    )
                source_excludes = _parse_nvfp4_excludes(config)
                quant_config = QuarkConfig._create_online_mxfp4_config(
                    model_type=hf_config.model_type,
                    source_excludes=source_excludes,
                    layer_quant_config=layer_quant_config,
                    packed_modules_mapping=config.get("packed_modules_mapping"),
                )
                # Excluded layers are kept as-is. When the base checkpoint is
                # FP8-serialized (e.g. DeepSeek-V4-Pro-NVFP4: FP8 attn/
                # shared_experts, NVFP4 routed experts) they load through FP8;
                # `weight_block_size` selects block vs per-tensor. Pure
                # NVFP4/ModelOpt-mixed sources keep excluded layers in bf16, and
                # their FP8 layers (if any) are enumerated in the layer map.
                excluded_fp8_config = _build_excluded_fp8_config(config)

View on GitHub (pinned to 0132848349)

Solutions

Drop --quantization quark_mxfp4 and let sglang load the model with its native (mixed FP8) quark quantization.
If you actually want MXFP4, start from a checkpoint that contains NVFP4 layers (or re-export one).
Add a pre-flight check on the checkpoint config for NVFP4 groups before enabling the flag (see validation code).

Example fix

# before
python -m sglang.launch_server --model granite_fp8_mixed --quantization quark_mxfp4  # raises

# after
python -m sglang.launch_server --model granite_fp8_mixed  # native mixed FP8 quark path

Defensive patterns

Strategy: validation

Validate before calling

layer_map = collect_layer_algos(config["config_groups"])
has_nvfp4 = any(a in ("NVFP4", "W4A16_NVFP4") for algos in layer_map.values() for a in algos)
if not has_nvfp4:
    raise RuntimeError("No NVFP4 layers; serve natively without --quantization quark_mxfp4")

Type guard

def has_nvfp4_layers(config: dict) -> bool:
    groups = config.get("config_groups", {})
    return any("fp4" in str(g).lower() for g in groups.values())

Try / catch

try:
    QuarkConfig.from_config(quant_config=config, hf_config=hf_config)
except NotImplementedError as e:
    if "no NVFP4 layers" in str(e):
        relaunch_without_quark_mxfp4_flag(model_path)
    raise

Prevention

Don't hardcode --quantization quark_mxfp4; detect NVFP4 presence from the quant config first.
Keep per-model launch configs annotated with the intended quantization path.
Add a preflight lint that maps checkpoint -> supported launch flags.

When it happens

Trigger: --quantization quark_mxfp4 on a MIXED_PRECISION checkpoint where _build_mixed_precision_layer_quant_config finds no group with algo NVFP4/W4A16_NVFP4 — e.g. an FP8-only mixed checkpoint.

Common situations: Users applying the quark_mxfp4 flag generically to every quark/Granite model, including FP8 mixed ones; scripts that hardcode --quantization quark_mxfp4 regardless of checkpoint contents.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/e10b724524b765ae. Report an issue: GitHub.