sgl-project/sglang · critical · RuntimeError

DeepSeek-V4 FP4 experts require torch.float4_e2m1fn_x2 suppo

Error message

DeepSeek-V4 FP4 experts require torch.float4_e2m1fn_x2 support.

What it means

Raised by _require_fp4_dtype when torch.float4_e2m1fn_x2 does not exist in the installed PyTorch. The DeepSeek-V4 FP4 expert path needs native FP4 dtype support (added in newer PyTorch nightly/2.8+ releases) to build and run FP4 quantized expert weights. It fails at weight post-processing, before any kernel launch.

Source

Thrown at python/sglang/srt/layers/quantization/fp8.py:137

_is_gfx95_supported = is_gfx95_supported()
# gfx942 (MI300) has no MX matmul HW; MXFP8 checkpoints are converted to
# block-fp8 [128,128] at load and run through the native block-fp8 kernels.
# SGLANG_FORCE_MXFP8_BLOCK_CONVERT=1 opts into that same block-fp8 path on
# gfx950 (MI35x): it routes the fp8 GEMMs / fused MoE through the mature aiter
# block-scale kernels instead of the native MX dot_scaled path (measured +20%
# throughput at equal accuracy on MiniMax-M3, GSM8K 0.9719 vs 0.9689).
_mxfp8_to_block_fp8_required = mxfp8_block_convert_required() or get_bool_env_var(
    "SGLANG_FORCE_MXFP8_BLOCK_CONVERT"
)
_use_hip_int4 = get_bool_env_var("SGLANG_INT4_WEIGHT") and _is_hip
_use_aiter = envs.SGLANG_USE_AITER.get() and _is_hip
_is_shuffle_moe_mxfp4 = is_gfx95_supported()


def _require_fp4_dtype():
    fp4_dtype = getattr(torch, "float4_e2m1fn_x2", None)
    if fp4_dtype is None:
        raise RuntimeError(
            "DeepSeek-V4 FP4 experts require torch.float4_e2m1fn_x2 support."
        )
    return fp4_dtype


if _use_aiter or _use_hip_int4:
    from aiter.ops.shuffle import shuffle_scale, shuffle_weight

if _use_aiter:
    from sglang.srt.layers.quantization.fp8_utils import (
        aiter_w8a8_block_fp8_linear,
        use_aiter_triton_gemm_w8a8_tuned_gfx950,
    )


ACTIVATION_SCHEMES = ["static", "dynamic"]

logger = logging.getLogger(__name__)

View on GitHub (pinned to 0132848349)

Solutions

Upgrade PyTorch to a build that defines torch.float4_e2m1fn_x2 (torch >= 2.8 or recent nightly): pip install -U torch --index-url https://download.pytorch.org/whl/nightly/cu128
If the checkpoint offers a non-FP4 variant, load that instead
Verify with python -c "import torch; print(hasattr(torch,'float4_e2m1fn_x2'))" before launching

Example fix

# before
torch.__version__  # 2.6 — model load crashes with RuntimeError
# after
pip install -U "torch>=2.8"
python -c "import torch; assert hasattr(torch, 'float4_e2m1fn_x2')"

Defensive patterns

Strategy: validation

Validate before calling

import torch
if not hasattr(torch, "float4_e2m1fn_x2"):
    raise SystemExit(f"torch {torch.__version__} lacks FP4 dtype; upgrade to >=2.8 for DeepSeek-V4 FP4")

Type guard

def has_fp4_dtype() -> bool:
    return getattr(torch, "float4_e2m1fn_x2", None) is not None

Prevention

Gate FP4-expert model launches on hasattr(torch, 'float4_e2m1fn_x2')
Pin torch>=2.8 in requirements when using FP4 checkpoints
Add a startup env check in serving scripts

When it happens

Trigger: process_weights_after_loading_block_quant or maybe_get_hip_aiter_quant_info hitting the FP4 expert path (DeepSeek-V4 FP4 checkpoints) on a PyTorch build lacking torch.float4_e2m1fn_x2 — i.e. torch < 2.8 or a stable release without FP4 dtype.

Common situations: Running a DeepSeek-V4 FP4 expert model on an older torch pinned by another framework; CI images with stale torch; environments where torch was downgraded for vLLM/other tool compatibility.

Related errors

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/b28ec72109dfe879. Report an issue: GitHub.