{"record":{"id":"60e390c25cecb03d","repo":"sgl-project/sglang","slug":"can-t-import-trtllm-fp8-block-scale-moe-from-flash","errorCode":null,"errorMessage":"Can't import trtllm_fp8_block_scale_moe from flashinfer. Please check flashinfer version.","messagePattern":"Can't import trtllm_fp8_block_scale_moe from flashinfer\\. Please check flashinfer version\\.","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"python/sglang/srt/layers/moe/flashinfer_trtllm_moe.py","lineNumber":76,"sourceCode":"    top_k: int,\n    n_group: Optional[int],\n    topk_group: Optional[int],\n    intermediate_size: int,\n    local_expert_offset: int,\n    local_num_experts: int,\n    routed_scaling_factor: Optional[float],\n    routing_method_type: int = 0,\n    use_shuffled_weight: bool = False,\n    weight_layout: int = 0,\n    enable_pdl: Optional[bool] = None,\n    tune_max_num_tokens: int = 8192,\n    fp8_quantization_type: Optional[int] = None,\n    activation_type: Optional[int] = None,\n) -> None:\n    try:\n        from flashinfer.fused_moe import trtllm_fp8_block_scale_moe\n    except ImportError as e:\n        raise ImportError(\n            \"Can't import trtllm_fp8_block_scale_moe from flashinfer. \"\n            \"Please check flashinfer version.\"\n        ) from e\n\n    kwargs = {\n        \"routing_logits\": routing_logits,\n        \"routing_bias\": routing_bias,\n        \"hidden_states\": hidden_states,\n        \"hidden_states_scale\": hidden_states_scale,\n        \"gemm1_weights\": gemm1_weights,\n        \"gemm1_weights_scale\": gemm1_weights_scale,\n        \"gemm1_alpha\": gemm1_alpha,\n        \"gemm1_beta\": gemm1_beta,\n        \"gemm1_clamp_limit\": gemm1_clamp_limit,\n        \"gemm2_weights\": gemm2_weights,\n        \"gemm2_weights_scale\": gemm2_weights_scale,\n        \"output\": output,\n        \"num_experts\": num_experts,","sourceCodeStart":58,"sourceCodeEnd":94,"githubUrl":"https://github.com/sgl-project/sglang/blob/0132848349585cfe6aae51c4941cbae872505f8a/python/sglang/srt/layers/moe/flashinfer_trtllm_moe.py#L58-L94","documentation":"Lazy import of trtllm_fp8_block_scale_moe from flashinfer.fused_moe failed, so the TRT-LLM fp8 block-scale MoE wrapper cannot run. This is a flashinfer version issue: the function exists only in newer flashinfer releases.","triggerScenarios":"Calling trtllm_fp8_block_scale_moe_out_wrapper (e.g. via fused_experts_none_to_flashinfer_trtllm_fp8) with an old or partial flashinfer install that lacks the symbol.","commonSituations":"Old flashinfer pinned in the environment; flashinfer installed from a stale wheel or source checkout; CUDA/arch mismatch causing the fused_moe module to fail importing its compiled extensions (surfacing as ImportError).","solutions":["Upgrade flashinfer to a release that includes trtllm_fp8_block_scale_moe (e.g. pip install -U flashinfer-python)","Verify the import manually: python -c \"from flashinfer.fused_moe import trtllm_fp8_block_scale_moe\" to see the underlying error","If the underlying failure is a compiled-extension/CUDA issue, reinstall flashinfer matching your torch and CUDA versions","Fall back to another fp8 MoE backend (triton) while flashinfer is unavailable"],"exampleFix":"# before\npip install flashinfer-python==0.1.*  # lacks symbol\n# after\npip install -U flashinfer-python\npython -c \"from flashinfer.fused_moe import trtllm_fp8_block_scale_moe\"","handlingStrategy":"fallback","validationCode":"try:\n    from flashinfer.fused_moe import trtllm_fp8_block_scale_moe  # noqa\n    HAS_TRTLLM_MOE = True\nexcept ImportError:\n    HAS_TRTLLM_MOE = False\nif not HAS_TRTLLM_MOE:\n    raise RuntimeError('flashinfer too old for trtllm fp8 MoE; upgrade or use triton backend')","typeGuard":null,"tryCatchPattern":"try:\n    trtllm_fp8_block_scale_moe_out_wrapper(...)\nexcept ImportError as e:\n    if 'trtllm_fp8_block_scale_moe' in str(e):\n        use_triton_fused_experts(...)  # fallback backend\n    else:\n        raise","preventionTips":["Pin a flashinfer version >= the one introducing the symbol","Smoke-test flashinfer imports at server startup","Match flashinfer build to torch/CUDA versions"],"tags":["flashinfer","trtllm","moe","fp8","import-error","version-mismatch"],"backgroundTag":"missing-optional-dependency","analyzedSha":"0132848349585cfe6aae51c4941cbae872505f8a","analyzedAt":"2026-08-28T05:10:05.995Z","schemaVersion":2},"datasetVersion":"2026-08-28T06:17:29.519Z"}