huggingface/candle · error

no cpu support for flash-attn

Error message

no cpu support for flash-attn

What it means

Raised by the `cpu_fwd` fallback of flash-attn's op in candle-flash-attn/src/lib.rs whenever flash-attention is invoked on CPU tensors. Flash-attention v2 is only implemented as a CUDA kernel; CPU execution is not supported, so running the attention on CPU inputs (e.g. via `softmax(Q@K^T)@V` manually or moving to GPU) is required instead.

Source

Thrown at candle-flash-attn/src/lib.rs:237

        Ok((dst, out_shape))
    }
}

impl candle::CustomOp3 for FlashAttn {
    fn name(&self) -> &'static str {
        "flash-attn"
    }

    fn cpu_fwd(
        &self,
        _: &CpuStorage,
        _: &Layout,
        _: &CpuStorage,
        _: &Layout,
        _: &CpuStorage,
        _: &Layout,
    ) -> Result<(CpuStorage, Shape)> {
        candle::bail!("no cpu support for flash-attn")
    }

    fn cuda_fwd(
        &self,
        q: &candle::CudaStorage,
        q_l: &Layout,
        k: &candle::CudaStorage,
        k_l: &Layout,
        v: &candle::CudaStorage,
        v_l: &Layout,
    ) -> Result<(candle::CudaStorage, Shape)> {
        match q.dtype() {
            candle::DType::F16 => self.cuda_fwd_t::<f16>(q, q_l, k, k_l, v, v_l, false),
            candle::DType::BF16 => self.cuda_fwd_t::<bf16>(q, q_l, k, k_l, v, v_l, true),
            dt => candle::bail!("flash-attn is only supported for f16/bf16 ({dt:?})"),
        }
    }
}

View on GitHub (pinned to d5fee525bf)

Solutions

1. Use CUDA: load the model on an NVIDIA GPU device (candle_nvcc_wrapper/cuda feature enabled)
2. On non-CUDA hardware replace the flash-attn op with standard candle nn::Sdpa / softmax-based attention
3. Never expect a CPU fallback; gate usage behind a CUDA-capable device check

Example fix

// before
let dev = Device::Cpu;
let attn = FlashAttnV2 { softmax_scale };
// after
let dev = Device::new_cuda(0)?; // requires cuda feature and NVIDIA GPU
// or on CPU-only machines use candle_nn::Sdpa instead of FlashAttn

let dev = q.device();
if !dev.is_cuda() {
    // use standard attention instead of flash-attn
    let attn = candle_nn::Sdpa { .. }; 
}

fn supports_flash_attn(t: &Tensor) -> bool {
    t.device().is_cuda() && matches!(t.dtype(), candle::DType::F16 | candle::DType::BF16)
}

let out = match flash_attn(&q, &k, &v, None, scale, causal) {
    Ok(o) => o,
    Err(e) if e.to_string().contains("no cpu support") || e.to_string().contains("only supported for f16") => {
        sdpa_fallback(&q, &k, &v, scale, causal)?
    }
    Err(e) => return Err(e),
};

When it happens

Trigger: Running a model that calls candle_flash_attn::flash_attn / FlashAttn while tensors (or the whole model) are on Device::Cpu; calling the op during a CPU fallback path or in tests without a GPU.

Common situations: Running examples on a machine without NVIDIA GPU, accidentally keeping some tensors on CPU so the op dispatches to cpu_fwd, CI environments without CUDA support.

Related errors

• compile with '--features flash-attn'
• compile with '--features flash-attn'
• compile with '--features flash-attn'
• compile with '--features flash-attn'
• compile with '--features flash-attn'

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/fb7edfaf40bd809f. Report an issue: GitHub.