{"record":{"id":"1e26a51b1d1fbb78","repo":"vllm-project/vllm","slug":"the-fused-grouped-topk-kernel-is-only-available-on","errorCode":null,"errorMessage":"The fused grouped_topk kernel is only available on CUDA platforms","messagePattern":"The fused grouped_topk kernel is only available on CUDA platforms","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"vllm/_custom_ops.py","lineNumber":2494,"sourceCode":"    routed_scaling_factor: float,\n    bias: torch.Tensor,\n    scoring_func: int = 0,\n):\n    \"\"\"\n    Perform grouped top-k routing for mixture of experts.\n\n    Args:\n        scores: Raw inputs (logits if scoring_func=1, scores if scoring_func=0)\n        num_expert_group: Number of expert groups\n        topk_group: Number of groups to select\n        topk: Number of experts to select per token\n        renormalize: Whether to renormalize the output weights\n        routed_scaling_factor: Scaling factor for routing weights\n        bias: Bias tensor (e_score_correction_bias). Always fused in kernel.\n        scoring_func: 0=none (no activation), 1=sigmoid\n    \"\"\"\n    if not current_platform.is_cuda():\n        raise NotImplementedError(\n            \"The fused grouped_topk kernel is only available on CUDA platforms\"\n        )\n    return torch.ops._moe_C.grouped_topk(\n        scores,\n        num_expert_group,\n        topk_group,\n        topk,\n        renormalize,\n        routed_scaling_factor,\n        bias,\n        scoring_func,\n    )\n\n\ndef moe_wna16_marlin_gemm(\n    input: torch.Tensor,\n    output: torch.Tensor | None,\n    b_qweight: torch.Tensor,","sourceCodeStart":2476,"sourceCodeEnd":2512,"githubUrl":"https://github.com/vllm-project/vllm/blob/c794754062d49a8fdb63ab3c5215b488b865030c/vllm/_custom_ops.py#L2476-L2512","documentation":"The fused grouped_topk wrapper (DeepSeek-V3-style grouped expert routing with e_score_correction_bias) calls torch.ops._moe_C.grouped_topk, which only exists as a CUDA kernel. On any other platform the wrapper raises NotImplementedError instead of dispatching.","triggerScenarios":"Calling vllm._custom_ops grouped_topk(scores, num_expert_group, topk_group, topk, renormalize, routed_scaling_factor, bias, scoring_func) when current_platform.is_cuda() is False.","commonSituations":"Serving a grouped-routing MoE (DeepSeek-V2/V3, Qwen3-MoE with group routing) on CPU/XPU/ROCm builds; unit tests of the topk wrapper on CI runners without GPUs.","solutions":["Run on a CUDA machine with the full _moe_C extension built","Use the pure-PyTorch grouped_topk fallback (vllm.model_executor.layers.moe.topk, e.g. via --disable-custom-topk style config / selecting the non-fused path)","Ensure vLLM was installed with CUDA support (not a CPU wheel) if a GPU is actually present"],"exampleFix":null,"handlingStrategy":"fallback","validationCode":"from vllm.platforms import current_platform\nuse_fused = current_platform.is_cuda()","typeGuard":"def fused_grouped_topk_available() -> bool:\n    from vllm.platforms import current_platform\n    return current_platform.is_cuda()","tryCatchPattern":"try:\n    w, i = ops grouped_topk fused call\nexcept NotImplementedError:\n    from vllm.model_executor.layers.moe.topk import grouped_topk\n    w, i = grouped_topk(...)  # reference implementation","preventionTips":["Prefer the layer-level API which picks fused vs reference automatically","Guard fused-kernel calls with current_platform.is_cuda() in portable code"],"tags":["moe","topk","routing","cuda-only","platform-support"],"backgroundTag":null,"analyzedSha":"c794754062d49a8fdb63ab3c5215b488b865030c","analyzedAt":"2026-08-14T21:17:39.825Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}