vllm-project/vllm · error · NotImplementedError

The fused grouped_topk kernel is only available on CUDA plat

Error message

The fused grouped_topk kernel is only available on CUDA platforms

What it means

The fused grouped_topk wrapper (DeepSeek-V3-style grouped expert routing with e_score_correction_bias) calls torch.ops._moe_C.grouped_topk, which only exists as a CUDA kernel. On any other platform the wrapper raises NotImplementedError instead of dispatching.

Source

Thrown at vllm/_custom_ops.py:2494

    routed_scaling_factor: float,
    bias: torch.Tensor,
    scoring_func: int = 0,
):
    """
    Perform grouped top-k routing for mixture of experts.

    Args:
        scores: Raw inputs (logits if scoring_func=1, scores if scoring_func=0)
        num_expert_group: Number of expert groups
        topk_group: Number of groups to select
        topk: Number of experts to select per token
        renormalize: Whether to renormalize the output weights
        routed_scaling_factor: Scaling factor for routing weights
        bias: Bias tensor (e_score_correction_bias). Always fused in kernel.
        scoring_func: 0=none (no activation), 1=sigmoid
    """
    if not current_platform.is_cuda():
        raise NotImplementedError(
            "The fused grouped_topk kernel is only available on CUDA platforms"
        )
    return torch.ops._moe_C.grouped_topk(
        scores,
        num_expert_group,
        topk_group,
        topk,
        renormalize,
        routed_scaling_factor,
        bias,
        scoring_func,
    )


def moe_wna16_marlin_gemm(
    input: torch.Tensor,
    output: torch.Tensor | None,
    b_qweight: torch.Tensor,

View on GitHub (pinned to c794754062)

Solutions

1. Run on a CUDA machine with the full _moe_C extension built
2. Use the pure-PyTorch grouped_topk fallback (vllm.model_executor.layers.moe.topk, e.g. via --disable-custom-topk style config / selecting the non-fused path)
3. Ensure vLLM was installed with CUDA support (not a CPU wheel) if a GPU is actually present

from vllm.platforms import current_platform
use_fused = current_platform.is_cuda()

def fused_grouped_topk_available() -> bool:
    from vllm.platforms import current_platform
    return current_platform.is_cuda()

try:
    w, i = ops grouped_topk fused call
except NotImplementedError:
    from vllm.model_executor.layers.moe.topk import grouped_topk
    w, i = grouped_topk(...)  # reference implementation

When it happens

Trigger: Calling vllm._custom_ops grouped_topk(scores, num_expert_group, topk_group, topk, renormalize, routed_scaling_factor, bias, scoring_func) when current_platform.is_cuda() is False.

Common situations: Serving a grouped-routing MoE (DeepSeek-V2/V3, Qwen3-MoE with group routing) on CPU/XPU/ROCm builds; unit tests of the topk wrapper on CI runners without GPUs.

Related errors

• The optimized moe_wna16_gemm kernel is only available on CUD
• Expert parallelism load balancing is only supported on CUDA
• multimodal preprocessing error: {0}
• unexpected non-control output on coordinator path: {message}
• asymmetric int8 activation quantization is unsupported on XP

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/1e26a51b1d1fbb78. Report an issue: GitHub.