{"record":{"id":"29dda99edce26720","repo":"huggingface/candle","slug":"flash-attn-is-only-supported-for-f16-bf16-dt","errorCode":null,"errorMessage":"flash-attn is only supported for f16/bf16 ({dt:?})","messagePattern":"flash-attn is only supported for f16/bf16 \\((.+?)\\)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-flash-attn/src/lib.rs","lineNumber":252,"sourceCode":"        _: &CpuStorage,\n        _: &Layout,\n    ) -> Result<(CpuStorage, Shape)> {\n        candle::bail!(\"no cpu support for flash-attn\")\n    }\n\n    fn cuda_fwd(\n        &self,\n        q: &candle::CudaStorage,\n        q_l: &Layout,\n        k: &candle::CudaStorage,\n        k_l: &Layout,\n        v: &candle::CudaStorage,\n        v_l: &Layout,\n    ) -> Result<(candle::CudaStorage, Shape)> {\n        match q.dtype() {\n            candle::DType::F16 => self.cuda_fwd_t::<f16>(q, q_l, k, k_l, v, v_l, false),\n            candle::DType::BF16 => self.cuda_fwd_t::<bf16>(q, q_l, k, k_l, v, v_l, true),\n            dt => candle::bail!(\"flash-attn is only supported for f16/bf16 ({dt:?})\"),\n        }\n    }\n}\n\n/// Flash-attention v2 layer.\n///\n/// This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`.\n/// Multi-query and grouped-query attention are supported by using tensors k and v with fewer heads\n/// than q, the number of heads in k and v has to be divisible by the number of heads in q.\n///\n/// # Arguments\n///\n/// * `q` - Query tensor with shape `(batch, seq_len_q, num_heads_q, head_size)`.\n/// * `k` - Key tensor with shape `(batch, seq_len_kv, num_heads_kv, head_size)`.\n/// * `v` - Value tensor with shape `(batch, seq_len_kv, num_heads_kv, head_size)`.\n///\n/// The resulting tensor has dimensions `(batch, seq_len_q, num_heads_q, head_size)`.\npub fn flash_attn(","sourceCodeStart":234,"sourceCodeEnd":270,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-flash-attn/src/lib.rs#L234-L270","documentation":"Raised in `cuda_fwd` of candle-flash-attn/src/lib.rs when the query (and implicitly k/v) dtype is neither F16 nor BF16. The CUDA flash-attention kernel is only instantiated for half-precision types, so e.g. f32/f8 inputs must be cast to f16 or bf16 before calling flash-attn.","triggerScenarios":"Calling flash_attn / FlashAttnV2 with q (and by contract k/v) tensors of dtype f32 — typical when the model runs in full float32 precision, or when tensors were upcast by prior ops.","commonSituations":"Running a model with dtype f32 weights, forgetting to call .to_dtype(DType::BF16/F16) after loading f32 safetensors, converting a model port that keeps hidden states in f32 while using flash-attn layers.","solutions":["Convert q/k/v to DType::F16 or DType::BF16 with .to_dtype() before the call","Load the model in bf16/f16 (e.g. dtype from config) instead of f32","Fall back to standard attention (candle_nn::Sdpa) when running in f32"],"exampleFix":"// before\nlet (q, k, v) = (q.to_device(&dev)?, k.to_device(&dev)?, v.to_device(&dev)?);\nflash_attn(&q, &k, &v, None, scale, true)?\n// after\nlet dt = q.dtype();\nlet (q, k, v) = (\n    q.to_dtype(DType::BF16)?.to_device(&dev)?,\n    k.to_dtype(DType::BF16)?.to_device(&dev)?,\n    v.to_dtype(DType::BF16)?.to_device(&dev)?,\n);\nflash_attn(&q, &k, &v, None, scale, true)?","handlingStrategy":"validation","validationCode":"fn ensure_half(t: &Tensor, dev: &Device) -> candle::Result<Tensor> {\n    match t.dtype() {\n        candle::DType::F16 | candle::DType::BF16 => Ok(t.clone()),\n        _ => t.to_dtype(candle::DType::BF16)?.to_device(dev),\n    }\n}","typeGuard":"fn is_half(t: &Tensor) -> bool {\n    matches!(t.dtype(), candle::DType::F16 | candle::DType::BF16)\n}","tryCatchPattern":"match flash_attn(&q, &k, &v, None, scale, causal) {\n    Err(e) if e.to_string().contains(\"only supported for f16/bf16\") => {\n        let (q, k, v) = (q.to_dtype(DType::BF16)?, k.to_dtype(DType::BF16)?, v.to_dtype(DType::BF16)?);\n        flash_attn(&q, &k, &v, None, scale, causal)?\n    }\n    r => r?,\n}","preventionTips":["Load models in bf16/f16 when using flash-attn layers","Upcast/downcast q/k/v uniformly right before the call","Assert dtype in forward: debug_assert!(is_half(&q))"],"tags":["cuda","flash-attention","dtype","precision"],"backgroundTag":"unsupported-dtype","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}