sgl-project/sglang · error · ImportError

The package `amd-quark` is required to use MX-FP4 models. Pl

Error message

The package `amd-quark` is required to use MX-FP4 models. Please install it with `pip install amd-quark`.

What it means

dequant_mxfp4 needs AMD's quark library kernel (mx.dq_mxfp4) to dequantize MX-FP4 tensors back to floating point. If `amd-quark` is not installed, the import of quark.torch.kernel.mx fails and the error is raised with the install command chained from the original ImportError.

Source

Thrown at python/sglang/srt/layers/quantization/mxfp4.py:277

    return quant_tensor, InFlexData(), scale


def _dequant_mxfp4_fake(
    x: torch.Tensor, scale: torch.Tensor, float_dtype: torch.dtype
) -> torch.Tensor:
    return torch.empty(
        (*x.shape[:-1], x.shape[-1] * 2), dtype=float_dtype, device=x.device
    )


@register_custom_op(fake_impl=_dequant_mxfp4_fake)
def dequant_mxfp4(
    x: torch.Tensor, scale: torch.Tensor, float_dtype: torch.dtype
) -> torch.Tensor:
    try:
        from quark.torch.kernel import mx
    except ImportError as err:
        raise ImportError(
            "The package `amd-quark` is required to use "
            "MX-FP4 models. Please install it with `pip install "
            "amd-quark`."
        ) from err

    return mx.dq_mxfp4(x, scale, float_dtype)


@register_custom_op(out_shape="x")
def quant_dequant_mxfp4(
    x: torch.Tensor, scale_calculation_mode: str = "even"
) -> torch.Tensor:
    try:
        from quark.torch.kernel import mx
    except ImportError as err:
        raise ImportError(
            "The package `amd-quark` is required to use "
            "MX-FP4 models. Please install it with `pip install "

View on GitHub (pinned to 0132848349)

Solutions

1. pip install amd-quark in the serving environment
2. If on NVIDIA hardware, prefer an SGLang path that keeps MX-FP4 in kernel-native format (no dequant fallback) or use a checkpoint format your stack supports
3. Pin a working amd-quark version in requirements if a release regressed the quark.torch.kernel.mx import

Example fix

# before
y = dequant_mxfp4(x, scale, torch.bfloat16)  # ImportError
# after
# pip install amd-quark
y = dequant_mxfp4(x, scale, torch.bfloat16)

try:
    from quark.torch.kernel import mx  # noqa: F401
    HAS_QUARK = True
except ImportError:
    HAS_QUARK = False

if not HAS_QUARK:
    raise SystemExit("pip install amd-quark before dequantizing MX-FP4 weights")
y = dequant_mxfp4(x, scale, torch.bfloat16)

import importlib.util

def has_amd_quark() -> bool:
    return importlib.util.find_spec("quark") is not None

try:
    y = dequant_mxfp4(x, scale, dt)
except ImportError as e:
    if "amd-quark" in str(e):
        subprocess.check_call([sys.executable, "-m", "pip", "install", "amd-quark"])
        y = dequant_mxfp4(x, scale, dt)  # retry once after install
    else:
        raise

When it happens

Trigger: Calling mxfp4.dequant_mxfp4(x, scale, float_dtype) (directly or via a model whose checkpoint stores MX-FP4 weights that must be dequantized on load) in a Python environment without the amd-quark package — e.g. on an NVIDIA machine or a venv missing the wheel.

Common situations: Loading an MX-FP4 quantized model on a box where amd-quark was never installed; environments where the ROCm/AMD extras were skipped; CI images built for CUDA that still reference MX-FP4 checkpoints.

Related errors

• Online MXFP4 requantization from compressed-tensors NVFP4 ch
• MIXED_PRECISION layer group {tail!r} has inconsistent quant
• MIXED_PRECISION layer group {tail!r} uses unsupported quant
• MIXED_PRECISION checkpoint has no NVFP4 layers to requantize
• Online MXFP4 quantization for MoE layers requires an AMD ROC

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/0ff9d1bc61de98ac. Report an issue: GitHub.