{"record":{"id":"b28ec72109dfe879","repo":"sgl-project/sglang","slug":"deepseek-v4-fp4-experts-require-torch-float4-e2m1f","errorCode":null,"errorMessage":"DeepSeek-V4 FP4 experts require torch.float4_e2m1fn_x2 support.","messagePattern":"DeepSeek-V4 FP4 experts require torch\\.float4_e2m1fn_x2 support\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"python/sglang/srt/layers/quantization/fp8.py","lineNumber":137,"sourceCode":"_is_gfx95_supported = is_gfx95_supported()\n# gfx942 (MI300) has no MX matmul HW; MXFP8 checkpoints are converted to\n# block-fp8 [128,128] at load and run through the native block-fp8 kernels.\n# SGLANG_FORCE_MXFP8_BLOCK_CONVERT=1 opts into that same block-fp8 path on\n# gfx950 (MI35x): it routes the fp8 GEMMs / fused MoE through the mature aiter\n# block-scale kernels instead of the native MX dot_scaled path (measured +20%\n# throughput at equal accuracy on MiniMax-M3, GSM8K 0.9719 vs 0.9689).\n_mxfp8_to_block_fp8_required = mxfp8_block_convert_required() or get_bool_env_var(\n    \"SGLANG_FORCE_MXFP8_BLOCK_CONVERT\"\n)\n_use_hip_int4 = get_bool_env_var(\"SGLANG_INT4_WEIGHT\") and _is_hip\n_use_aiter = envs.SGLANG_USE_AITER.get() and _is_hip\n_is_shuffle_moe_mxfp4 = is_gfx95_supported()\n\n\ndef _require_fp4_dtype():\n    fp4_dtype = getattr(torch, \"float4_e2m1fn_x2\", None)\n    if fp4_dtype is None:\n        raise RuntimeError(\n            \"DeepSeek-V4 FP4 experts require torch.float4_e2m1fn_x2 support.\"\n        )\n    return fp4_dtype\n\n\nif _use_aiter or _use_hip_int4:\n    from aiter.ops.shuffle import shuffle_scale, shuffle_weight\n\nif _use_aiter:\n    from sglang.srt.layers.quantization.fp8_utils import (\n        aiter_w8a8_block_fp8_linear,\n        use_aiter_triton_gemm_w8a8_tuned_gfx950,\n    )\n\n\nACTIVATION_SCHEMES = [\"static\", \"dynamic\"]\n\nlogger = logging.getLogger(__name__)","sourceCodeStart":119,"sourceCodeEnd":155,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/quantization/fp8.py#L119-L155","documentation":"Raised by _require_fp4_dtype when torch.float4_e2m1fn_x2 does not exist in the installed PyTorch. The DeepSeek-V4 FP4 expert path needs native FP4 dtype support (added in newer PyTorch nightly/2.8+ releases) to build and run FP4 quantized expert weights. It fails at weight post-processing, before any kernel launch.","triggerScenarios":"process_weights_after_loading_block_quant or maybe_get_hip_aiter_quant_info hitting the FP4 expert path (DeepSeek-V4 FP4 checkpoints) on a PyTorch build lacking torch.float4_e2m1fn_x2 — i.e. torch < 2.8 or a stable release without FP4 dtype.","commonSituations":"Running a DeepSeek-V4 FP4 expert model on an older torch pinned by another framework; CI images with stale torch; environments where torch was downgraded for vLLM/other tool compatibility.","solutions":["Upgrade PyTorch to a build that defines torch.float4_e2m1fn_x2 (torch >= 2.8 or recent nightly): pip install -U torch --index-url https://download.pytorch.org/whl/nightly/cu128","If the checkpoint offers a non-FP4 variant, load that instead","Verify with python -c \"import torch; print(hasattr(torch,'float4_e2m1fn_x2'))\" before launching"],"exampleFix":"# before\ntorch.__version__  # 2.6 — model load crashes with RuntimeError\n# after\npip install -U \"torch>=2.8\"\npython -c \"import torch; assert hasattr(torch, 'float4_e2m1fn_x2')\"","handlingStrategy":"validation","validationCode":"import torch\nif not hasattr(torch, \"float4_e2m1fn_x2\"):\n    raise SystemExit(f\"torch {torch.__version__} lacks FP4 dtype; upgrade to >=2.8 for DeepSeek-V4 FP4\")","typeGuard":"def has_fp4_dtype() -> bool:\n    return getattr(torch, \"float4_e2m1fn_x2\", None) is not None","tryCatchPattern":null,"preventionTips":["Gate FP4-expert model launches on hasattr(torch, 'float4_e2m1fn_x2')","Pin torch>=2.8 in requirements when using FP4 checkpoints","Add a startup env check in serving scripts"],"tags":["quantization","fp4","pytorch-version","deepseek"],"backgroundTag":"missing-dtype-support","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}