{"record":{"id":"fb7edfaf40bd809f","repo":"huggingface/candle","slug":"no-cpu-support-for-flash-attn","errorCode":null,"errorMessage":"no cpu support for flash-attn","messagePattern":"no cpu support for flash-attn","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-flash-attn/src/lib.rs","lineNumber":237,"sourceCode":"        Ok((dst, out_shape))\n    }\n}\n\nimpl candle::CustomOp3 for FlashAttn {\n    fn name(&self) -> &'static str {\n        \"flash-attn\"\n    }\n\n    fn cpu_fwd(\n        &self,\n        _: &CpuStorage,\n        _: &Layout,\n        _: &CpuStorage,\n        _: &Layout,\n        _: &CpuStorage,\n        _: &Layout,\n    ) -> Result<(CpuStorage, Shape)> {\n        candle::bail!(\"no cpu support for flash-attn\")\n    }\n\n    fn cuda_fwd(\n        &self,\n        q: &candle::CudaStorage,\n        q_l: &Layout,\n        k: &candle::CudaStorage,\n        k_l: &Layout,\n        v: &candle::CudaStorage,\n        v_l: &Layout,\n    ) -> Result<(candle::CudaStorage, Shape)> {\n        match q.dtype() {\n            candle::DType::F16 => self.cuda_fwd_t::<f16>(q, q_l, k, k_l, v, v_l, false),\n            candle::DType::BF16 => self.cuda_fwd_t::<bf16>(q, q_l, k, k_l, v, v_l, true),\n            dt => candle::bail!(\"flash-attn is only supported for f16/bf16 ({dt:?})\"),\n        }\n    }\n}","sourceCodeStart":219,"sourceCodeEnd":255,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-flash-attn/src/lib.rs#L219-L255","documentation":"Raised by the `cpu_fwd` fallback of flash-attn's op in candle-flash-attn/src/lib.rs whenever flash-attention is invoked on CPU tensors. Flash-attention v2 is only implemented as a CUDA kernel; CPU execution is not supported, so running the attention on CPU inputs (e.g. via `softmax(Q@K^T)@V` manually or moving to GPU) is required instead.","triggerScenarios":"Running a model that calls candle_flash_attn::flash_attn / FlashAttn while tensors (or the whole model) are on Device::Cpu; calling the op during a CPU fallback path or in tests without a GPU.","commonSituations":"Running examples on a machine without NVIDIA GPU, accidentally keeping some tensors on CPU so the op dispatches to cpu_fwd, CI environments without CUDA support.","solutions":["Use CUDA: load the model on an NVIDIA GPU device (candle_nvcc_wrapper/cuda feature enabled)","On non-CUDA hardware replace the flash-attn op with standard candle nn::Sdpa / softmax-based attention","Never expect a CPU fallback; gate usage behind a CUDA-capable device check"],"exampleFix":"// before\nlet dev = Device::Cpu;\nlet attn = FlashAttnV2 { softmax_scale };\n// after\nlet dev = Device::new_cuda(0)?; // requires cuda feature and NVIDIA GPU\n// or on CPU-only machines use candle_nn::Sdpa instead of FlashAttn","handlingStrategy":"fallback","validationCode":"let dev = q.device();\nif !dev.is_cuda() {\n    // use standard attention instead of flash-attn\n    let attn = candle_nn::Sdpa { .. }; \n}","typeGuard":"fn supports_flash_attn(t: &Tensor) -> bool {\n    t.device().is_cuda() && matches!(t.dtype(), candle::DType::F16 | candle::DType::BF16)\n}","tryCatchPattern":"let out = match flash_attn(&q, &k, &v, None, scale, causal) {\n    Ok(o) => o,\n    Err(e) if e.to_string().contains(\"no cpu support\") || e.to_string().contains(\"only supported for f16\") => {\n        sdpa_fallback(&q, &k, &v, scale, causal)?\n    }\n    Err(e) => return Err(e),\n};","preventionTips":["Gate flash-attn usage behind a CUDA device check at model construction","Compile with cuda feature and run on NVIDIA GPU; otherwise pick Sdpa","Avoid partial device placement that lets the op dispatch to CPU"],"tags":["cpu","flash-attention","unsupported-platform","cuda-only"],"backgroundTag":"cpu-not-supported","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}