sgl-project/sglang · error · NotImplementedError

Online MXFP4 requantization from compressed-tensors NVFP4 ch

Error message

Online MXFP4 requantization from compressed-tensors NVFP4 checkpoints is not supported at this time.

What it means

In quark.py, _detect_nvfp4_source inspects the checkpoint's quant config to decide whether online MXFP4 requantization is possible. When quant_method is compressed-tensors/compressed_tensors, the weights may be NVFP4 but packed in a compressed-tensors layout that this online requantization path cannot consume, so it raises NotImplementedError rather than producing garbage.

Source

Thrown at python/sglang/srt/layers/quantization/quark/quark.py:116

        w0, w1 = weight
        is_nvfp4_weight = (
            isinstance(w0, dict)
            and w0.get("dtype") == "fp4"
            and w0.get("qscheme") == "per_group"
            and w0.get("group_size") == 16
            and not w0.get("is_dynamic")
        )
        is_nvfp4_scale_2 = (
            isinstance(w1, dict)
            and w1.get("dtype") == "fp8_e4m3"
            and w1.get("qscheme") == "per_tensor"
            and not w1.get("is_dynamic")
        )
        if is_nvfp4_weight and is_nvfp4_scale_2:
            return Nvfp4SourceConfig()
        return None
    if quant_method in ("compressed-tensors", "compressed_tensors"):
        raise NotImplementedError(
            "Online MXFP4 requantization from compressed-tensors NVFP4 "
            "checkpoints is not supported at this time."
        )
    return None


# Target quant specs used when synthesizing a per-layer config for a
# MIXED_PRECISION source. The MXFP4 spec is the online-requant target shape
# recognized by `_is_mx_fp4`; the FP8 spec is the per-tensor W8A8 shape
# recognized by `_is_fp8_w8a8` (no requantization).
_MXFP4_TARGET_SPEC: Dict[str, Any] = {
    "weight": {
        "dtype": "fp4",
        "qscheme": "per_group",
        "group_size": 32,
        "is_dynamic": False,
        "scale_format": "e8m0",
    },

View on GitHub (pinned to 0132848349)

Solutions

1. Serve the compressed-tensors NVFP4 checkpoint with its native method: use the model's own quant config (drop --quantization quark_mxfp4) so sglang's compressed-tensors NVFP4 path handles it.
2. Or re-export/convert the checkpoint to a quark-style quant config that the online requantization path supports.
3. Check for a newer sglang version where compressed-tensors NVFP4 online requantization may be supported.

Example fix

# before
python -m sglang.launch_server --model nvfp4_ct_model --quantization quark_mxfp4  # raises NotImplementedError

# after
python -m sglang.launch_server --model nvfp4_ct_model  # native compressed-tensors NVFP4 path

qm = config.get("quant_method")
if qm in ("compressed-tensors", "compressed_tensors") and requant_to == "quark_mxfp4":
    raise RuntimeError("Online MXFP4 requant of compressed-tensors NVFP4 not supported; serve natively")
QuarkConfig.from_config(quant_config=config, hf_config=hf_config, ...)

def supports_online_mxfp4(quant_method: str) -> bool:
    return quant_method == "quark"

try:
    cfg = QuarkConfig.from_config(quant_config=config, hf_config=hf_config)
except NotImplementedError as e:
    if "compressed-tensors" in str(e):
        cfg = None  # fall back to native compressed-tensors loading path
    else:
        raise

When it happens

Trigger: Loading a checkpoint whose quant config has quant_method "compressed-tensors" (or "compressed_tensors") with --quantization quark_mxfp4 / requantization into quark MXFP4; from_config -> _detect_nvfp4_source hits the compressed-tensors branch and raises.

Common situations: Exporting NVFP4 models via llm-compressor (which writes compressed-tensors configs) and then trying to serve them with sglang's quark_mxfp4 online requantization; assuming all NVFP4 checkpoints are interchangeable across quant frameworks.

Related errors

• MIXED_PRECISION checkpoint has no NVFP4 layers to requantize
• MIXED_PRECISION layer group {tail!r} has inconsistent quant
• MIXED_PRECISION layer group {tail!r} uses unsupported quant
• Requantization into {config['requantization_method']} is not
• The package `amd-quark` is required to use MX-FP4 models. Pl

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/00fe21413d19e3ac. Report an issue: GitHub.