sgl-project/sglang · error · NotImplementedError

Only block_quant=True is supported in Quark MXFP4 requantiza

Error message

Only block_quant=True is supported in Quark MXFP4 requantization, got block_quant=False.

What it means

FP8→MXFP4 requantization in QuarkW4A4MXFp4MoE requires the FP8 checkpoint to use block quantization (weight_block_size set). A per-tensor FP8 checkpoint has no per-block scales, so per-group MXFP4 requantization is impossible.

Source

Thrown at python/sglang/srt/layers/quantization/quark/schemes/quark_w4a4_mxfp4_moe.py:139

                    num_experts=num_experts,
                    hidden_size=hidden_size,
                    intermediate_size_per_partition=intermediate_size_per_partition,
                    original_weight_loader=original_weight_loader,
                    extra_weight_attrs=extra_weight_attrs,
                )
            elif isinstance(self.dequantization_config, Fp8Config):
                with_bias = extra_weight_attrs.pop("with_bias", False)
                self.with_bias = with_bias

                if self.dequantization_config.use_mxfp8:
                    raise NotImplementedError(
                        "use_mxfp8=True is not supported in Quark MXFP4 requantization."
                    )

                block_quant = self.dequantization_config.weight_block_size is not None

                if not block_quant:
                    raise NotImplementedError(
                        "Only block_quant=True is supported in Quark MXFP4 requantization, got block_quant=False."
                    )

                # `_fp8_loaded_numel` is used to trigger FP8 -> MXFP4 requantization once all weights are loaded.
                # `_fp8_materialized` is used to ensure only one thread materializes weights from meta device.
                layer._fp8_loaded_numel = 0
                layer._fp8_materialized = False
                layer._load_device = torch.get_default_device()
                layer._fp8_loading_lock = threading.Lock()

                # Custom weight loader handling FP8->MXFP4 conversion.
                fp8_to_mxfp4_weight_loader = self.get_online_fp8_to_mxfp4_weight_loader(
                    layer, original_weight_loader
                )

                extra_weight_attrs["weight_loader"] = fp8_to_mxfp4_weight_loader
                # Create FP8 MoE weight parameters on meta device to avoid device memory overhead during weight loading, as the resulting model uses MXFP4 using less device memory.
                # The weight loader handles progressive FP8 weight materialization on device.

View on GitHub (pinned to 0132848349)

Solutions

1. Use a block-quantized FP8 checkpoint (weight_block_size defined, typically [128,128])
2. Directly use an MXFP4-serialized checkpoint instead of requantizing
3. Re-export the model with quark using block-wise FP8 quantization

Example fix

// before
"dequantization_config": {"quant_type": "fp8", "weight_block_size": null}
// after
"dequantization_config": {"quant_type": "fp8", "weight_block_size": [128, 128]}

assert quant_config.dequantization_config.weight_block_size is not None, 'FP8->MXFP4 requant requires block-quantized FP8'

When it happens

Trigger: dequantization_config is Fp8Config with weight_block_size=None in create_weights of QuarkW4A4MXFp4MoE.

Common situations: Loading a per-tensor FP8 quark export into the MXFP4 MoE scheme; configs exported for a different kernel expecting per-tensor scales.

Related errors

• The output_size of gate's and up's weight = {intermediate_si
• The input_size of down's weight = {intermediate_size_per_par
• The hpc_ops MoE runner backend only supports FP8-quantized M
• For FP8 Fused MoE layer, we require either per tensor or cha
• Unsupported weight quantization strategy: {self.weight_quant

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/7f6a0ac0ce2ebeb8. Report an issue: GitHub.