hiyouga/LlamaFactory · error · RuntimeError

CudaFusedMoEKernel requires CUDA, current accelerator is {cu

Error message

CudaFusedMoEKernel requires CUDA, current accelerator is {current}.

What it means

The cuda_fused_moe kernel plugin replaces HuggingFace per-expert MoE loops with fused Triton kernels, which only exist for NVIDIA GPUs. Before applying, check_device() compares the current accelerator type to DeviceType.CUDA and raises RuntimeError if it differs (e.g. cpu, mlu, npu, xpu).

Source

Thrown at src/llamafactory/v1/plugins/model_plugins/kernels/ops/mlp/cuda_fused_moe.py:375

# ---------------------------------------------------------------------------


@KernelPlugin("cuda_fused_moe").register()
class CudaFusedMoEKernel(BaseKernel):
    """Pure-Triton fused MoE kernel for NVIDIA CUDA GPUs.

    Replaces HuggingFace per-expert Python loops with a fully fused Triton pipeline:
    - Forward: scatter + grouped GEMMs + gather (single kernel per GEMM)
    - Backward: all dX and dW via grouped GEMMs (no Python loops)

    Requires: CUDA GPU + Triton
    """

    @staticmethod
    def check_device() -> None:
        current = get_current_accelerator().type
        if current != DeviceType.CUDA:
            raise RuntimeError(f"CudaFusedMoEKernel requires CUDA, current accelerator is {current}.")

    @staticmethod
    def check_deps() -> None:
        if _TRITON_IMPORT_ERROR is not None:
            raise RuntimeError("cuda_fused_moe requires Triton.") from _TRITON_IMPORT_ERROR

    @staticmethod
    def _apply(**kwargs) -> HFModel:
        model = kwargs.get("model")

        archs = getattr(model.config, "architectures", None) or []
        target_mapping = None
        for arch in archs:
            if arch in _TRITON_MOE_MAPPING:
                target_mapping = _TRITON_MOE_MAPPING[arch]
                break

        if target_mapping is None:

View on GitHub (pinned to f28afaf635)

Solutions

  1. Remove cuda_fused_moe from the kernels list on non-CUDA hardware
  2. Or guard the kernel list per node, selecting npu_fused_moe on Ascend and cuda_fused_moe on NVIDIA
  3. Run on a CUDA GPU if the fused MoE path is required
  4. If you expected CUDA, verify torch detects the GPU (nvidia-smi, torch.cuda.is_available) and that get_current_accelerator().type reports cuda

Example fix

# before
kernels: [cuda_fused_moe]  # fails on CPU/NPU nodes

# after
kernels: {cuda: [cuda_fused_moe], npu: [npu_fused_moe]}  # select per device
Defensive patterns

Strategy: validation

Validate before calling

from llamafactory.accelerator import get_current_accelerator
if get_current_accelerator().type != "cuda":
    kernels = [k for k in kernels if k != "cuda_fused_moe"]

Prevention

When it happens

Trigger: Registering/applying the cuda_fused_moe kernel plugin on a machine whose accelerator is not CUDA — CPU-only box, Apple silicon, or an NPU/MLU device — or before the accelerator was initialized so the type resolves to something unexpected.

Common situations: Sharing a config with kernels: [cuda_fused_moe] across heterogeneous clusters; running a smoke test or data-prep job on CPU with the full kernel list enabled; running on Ascend/Cambricon hardware with a CUDA-targeted config.

Related errors

AI-assisted analysis of hiyouga/LlamaFactory@f28afaf635 (2026-08-14). Data as JSON: /api/errors/68b46a654995f6a0. Report an issue: GitHub.