sgl-project/sglang · error · NotImplementedError

use_mxfp8=True is not supported in Quark MXFP4 requantizatio

Error message

use_mxfp8=True is not supported in Quark MXFP4 requantization.

What it means

When requantizing an FP8-serialized checkpoint into MXFP4 inside QuarkW4A4MXFp4MoE.create_weights, the dequantization config is Fp8Config but with use_mxfp8=True. Only plain FP8 checkpoints can be requantized to MXFP4, not already-MXFP8 ones.

Source

Thrown at python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py:132

        # Handle source-checkpoint -> MXFP4 requantization at load time. The
        # source may be NVFP4 (ModelOpt/Quark) or FP8 (block-quantized).
        if self.dequantization_config is not None:
            if isinstance(self.dequantization_config, Nvfp4SourceConfig):
                self._create_weights_from_nvfp4_moe(
                    layer=layer,
                    num_experts=num_experts,
                    hidden_size=hidden_size,
                    intermediate_size_per_partition=intermediate_size_per_partition,
                    original_weight_loader=original_weight_loader,
                    extra_weight_attrs=extra_weight_attrs,
                )
            elif isinstance(self.dequantization_config, Fp8Config):
                with_bias = extra_weight_attrs.pop("with_bias", False)
                self.with_bias = with_bias

                if self.dequantization_config.use_mxfp8:
                    raise NotImplementedError(
                        "use_mxfp8=True is not supported in Quark MXFP4 requantization."
                    )

                block_quant = self.dequantization_config.weight_block_size is not None

                if not block_quant:
                    raise NotImplementedError(
                        "Only block_quant=True is supported in Quark MXFP4 requantization, got block_quant=False."
                    )

                # `_fp8_loaded_numel` is used to trigger FP8 -> MXFP4 requantization once all weights are loaded.
                # `_fp8_materialized` is used to ensure only one thread materializes weights from meta device.
                layer._fp8_loaded_numel = 0
                layer._fp8_materialized = False
                layer._load_device = torch.get_default_device()
                layer._fp8_loading_lock = threading.Lock()

                # Custom weight loader handling FP8->MXFP4 conversion.

View on GitHub (pinned to 0132848349)

Solutions

1. Export the model with plain FP8 (use_mxfp8=False) weights so requantization to MXFP4 is possible
2. Use an MXFP4-serialized checkpoint directly instead of requantizing
3. Use the appropriate scheme for MXFP8 checkpoints instead of QuarkW4A4MXFp4MoE

Example fix

// before
"quant_config": {"dequantization_config": {"quant_type": "fp8", "use_mxfp8": true}}
// after
"quant_config": {"dequantization_config": {"quant_type": "fp8", "use_mxfp8": false}}

dq = quant_config.dequantization_config
assert not (type(dq).__name__ == 'Fp8Config' and getattr(dq, 'use_mxfp8', False)), 'use_mxfp8 checkpoints cannot be requantized to MXFP4'

def is_requantizable_fp8(dq):
    return type(dq).__name__ == 'Fp8Config' and not getattr(dq, 'use_mxfp8', False) and dq.weight_block_size is not None

When it happens

Trigger: Quark quant config whose dequantization_config is Fp8Config with use_mxfp8=True loaded by QuarkW4A4MXFp4MoE.create_weights.

Common situations: Mixing an MXFP8 AMD checkpoint with an MXFP4 requant path; hand-edited quant config; newer quark export setting use_mxfp8 by default.

Related errors

• The package `amd-quark` is required to use MX-FP4 models. Pl
• Online MXFP4 requantization from compressed-tensors NVFP4 ch
• MIXED_PRECISION layer group {tail!r} has inconsistent quant
• MIXED_PRECISION layer group {tail!r} uses unsupported quant
• MIXED_PRECISION checkpoint has no NVFP4 layers to requantize

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/fe191ad703cd94ee. Report an issue: GitHub.