sgl-project/sglang · error · ImportError

Can't import trtllm_fp8_block_scale_moe from flashinfer. Ple

Error message

Can't import trtllm_fp8_block_scale_moe from flashinfer. Please check flashinfer version.

What it means

Lazy import of trtllm_fp8_block_scale_moe from flashinfer.fused_moe failed, so the TRT-LLM fp8 block-scale MoE wrapper cannot run. This is a flashinfer version issue: the function exists only in newer flashinfer releases.

Source

Thrown at python/sglang/srt/layers/moe/flashinfer_trtllm_moe.py:76

    top_k: int,
    n_group: Optional[int],
    topk_group: Optional[int],
    intermediate_size: int,
    local_expert_offset: int,
    local_num_experts: int,
    routed_scaling_factor: Optional[float],
    routing_method_type: int = 0,
    use_shuffled_weight: bool = False,
    weight_layout: int = 0,
    enable_pdl: Optional[bool] = None,
    tune_max_num_tokens: int = 8192,
    fp8_quantization_type: Optional[int] = None,
    activation_type: Optional[int] = None,
) -> None:
    try:
        from flashinfer.fused_moe import trtllm_fp8_block_scale_moe
    except ImportError as e:
        raise ImportError(
            "Can't import trtllm_fp8_block_scale_moe from flashinfer. "
            "Please check flashinfer version."
        ) from e

    kwargs = {
        "routing_logits": routing_logits,
        "routing_bias": routing_bias,
        "hidden_states": hidden_states,
        "hidden_states_scale": hidden_states_scale,
        "gemm1_weights": gemm1_weights,
        "gemm1_weights_scale": gemm1_weights_scale,
        "gemm1_alpha": gemm1_alpha,
        "gemm1_beta": gemm1_beta,
        "gemm1_clamp_limit": gemm1_clamp_limit,
        "gemm2_weights": gemm2_weights,
        "gemm2_weights_scale": gemm2_weights_scale,
        "output": output,
        "num_experts": num_experts,

View on GitHub (pinned to 0132848349)

Solutions

1. Upgrade flashinfer to a release that includes trtllm_fp8_block_scale_moe (e.g. pip install -U flashinfer-python)
2. Verify the import manually: python -c "from flashinfer.fused_moe import trtllm_fp8_block_scale_moe" to see the underlying error
3. If the underlying failure is a compiled-extension/CUDA issue, reinstall flashinfer matching your torch and CUDA versions
4. Fall back to another fp8 MoE backend (triton) while flashinfer is unavailable

Example fix

# before
pip install flashinfer-python==0.1.*  # lacks symbol
# after
pip install -U flashinfer-python
python -c "from flashinfer.fused_moe import trtllm_fp8_block_scale_moe"

try:
    from flashinfer.fused_moe import trtllm_fp8_block_scale_moe  # noqa
    HAS_TRTLLM_MOE = True
except ImportError:
    HAS_TRTLLM_MOE = False
if not HAS_TRTLLM_MOE:
    raise RuntimeError('flashinfer too old for trtllm fp8 MoE; upgrade or use triton backend')

try:
    trtllm_fp8_block_scale_moe_out_wrapper(...)
except ImportError as e:
    if 'trtllm_fp8_block_scale_moe' in str(e):
        use_triton_fused_experts(...)  # fallback backend
    else:
        raise

When it happens

Trigger: Calling trtllm_fp8_block_scale_moe_out_wrapper (e.g. via fused_experts_none_to_flashinfer_trtllm_fp8) with an old or partial flashinfer install that lacks the symbol.

Common situations: Old flashinfer pinned in the environment; flashinfer installed from a stale wheel or source checkout; CUDA/arch mismatch causing the fused_moe module to fail importing its compiled extensions (surfacing as ImportError).

Related errors

• Can't import trtllm_fp8_block_scale_routed_moe from flashinf
• Kimi-K3 DCP with decode_attention_backend='cutedsl_mla' requ
• Failed to import 'set_transfer_engine' from 'mooncake.pg'. P
• Unsupported cute dtype {input.dtype}
• CuteDSL masked MoE supports activation 'silu' (gated) or 're

AI-assisted analysis of sgl-project/sglang@0132848349 (2026-08-28). Data as JSON: /api/errors/60e390c25cecb03d. Report an issue: GitHub.