vllm-project/vllm · error · NotImplementedError

The optimized moe_wna16_gemm kernel is only available on CUD

Error message

The optimized moe_wna16_gemm kernel is only available on CUDA platforms

What it means

moe_wna16_gemm dispatches directly to torch.ops._moe_C.moe_wna16_gemm, a compiled CUDA extension for weight-only (WnA16, e.g. W4A16/W8A16) grouped MoE GEMM. There is no CPU/XPU/ROCm implementation, so the Python wrapper raises NotImplementedError before touching the extension when current_platform.is_cuda() is False.

Source

Thrown at vllm/_custom_ops.py:2341

def moe_wna16_gemm(
    input: torch.Tensor,
    output: torch.Tensor,
    b_qweight: torch.Tensor,
    b_scales: torch.Tensor,
    b_qzeros: torch.Tensor | None,
    topk_weights: torch.Tensor | None,
    sorted_token_ids: torch.Tensor,
    experts_ids: torch.Tensor,
    num_tokens_post_pad: torch.Tensor,
    top_k: int,
    BLOCK_SIZE_M: int,
    BLOCK_SIZE_N: int,
    BLOCK_SIZE_K: int,
    bit: int,
) -> torch.Tensor:
    if not current_platform.is_cuda():
        raise NotImplementedError(
            "The optimized moe_wna16_gemm kernel is only available on CUDA platforms"
        )
    torch.ops._moe_C.moe_wna16_gemm(
        input,
        output,
        b_qweight,
        b_scales,
        b_qzeros,
        topk_weights,
        sorted_token_ids,
        experts_ids,
        num_tokens_post_pad,
        top_k,
        BLOCK_SIZE_M,
        BLOCK_SIZE_N,
        BLOCK_SIZE_K,
        bit,
    )

View on GitHub (pinned to c794754062)

Solutions

1. Run on a CUDA GPU with a CUDA-enabled vLLM build
2. Switch to a quantization format whose MoE path has kernels for your platform (e.g. unquantized or a marlin-compatible format)
3. In tests, skip or mock this wrapper when torch.cuda.is_available() is False

from vllm.platforms import current_platform
assert current_platform.is_cuda(), "moe_wna16_gemm requires CUDA"

def moe_wna16_available() -> bool:
    from vllm.platforms import current_platform
    return current_platform.is_cuda()

if not moe_wna16_available():
    out = torch_reference_moe_gemm(...)  # fallback impl
else:
    out = ops.moe_wna16_gemm(...)

When it happens

Trigger: Calling vllm._custom_ops.moe_wna16_gemm(...) on a non-CUDA platform, or in an environment where the CUDA platform detector does not identify as CUDA (CPU-only build, XPU, etc.).

Common situations: Unit-testing MoE kernel wrappers on a CPU machine; running an int4/int8 weight-only MoE model (e.g. Qwen/Mixtral GPTQ variants) with VLLM_PLATFORM set to a non-CUDA backend.

Related errors

• The fused grouped_topk kernel is only available on CUDA plat
• asymmetric int8 activation quantization is unsupported on XP
• Expert parallelism load balancing is only supported on CUDA
• chat request must contain at least one message
• multimodal preprocessing error: {0}

AI-assisted analysis of vllm-project/vllm@c794754062 (2026-08-14). Data as JSON: /api/errors/5165eb83a20e18df. Report an issue: GitHub.