huggingface/candle · error

flash-attn is only supported for f16/bf16 ({dt:?})

Error message

flash-attn is only supported for f16/bf16 ({dt:?})

What it means

Raised in `cuda_fwd` of candle-flash-attn/src/lib.rs when the query (and implicitly k/v) dtype is neither F16 nor BF16. The CUDA flash-attention kernel is only instantiated for half-precision types, so e.g. f32/f8 inputs must be cast to f16 or bf16 before calling flash-attn.

Source

Thrown at candle-flash-attn/src/lib.rs:252

        _: &CpuStorage,
        _: &Layout,
    ) -> Result<(CpuStorage, Shape)> {
        candle::bail!("no cpu support for flash-attn")
    }

    fn cuda_fwd(
        &self,
        q: &candle::CudaStorage,
        q_l: &Layout,
        k: &candle::CudaStorage,
        k_l: &Layout,
        v: &candle::CudaStorage,
        v_l: &Layout,
    ) -> Result<(candle::CudaStorage, Shape)> {
        match q.dtype() {
            candle::DType::F16 => self.cuda_fwd_t::<f16>(q, q_l, k, k_l, v, v_l, false),
            candle::DType::BF16 => self.cuda_fwd_t::<bf16>(q, q_l, k, k_l, v, v_l, true),
            dt => candle::bail!("flash-attn is only supported for f16/bf16 ({dt:?})"),
        }
    }
}

/// Flash-attention v2 layer.
///
/// This implements scaled dot-product attention, `softmax(Q @ K^T . softmax_scale) @ V`.
/// Multi-query and grouped-query attention are supported by using tensors k and v with fewer heads
/// than q, the number of heads in k and v has to be divisible by the number of heads in q.
///
/// # Arguments
///
/// * `q` - Query tensor with shape `(batch, seq_len_q, num_heads_q, head_size)`.
/// * `k` - Key tensor with shape `(batch, seq_len_kv, num_heads_kv, head_size)`.
/// * `v` - Value tensor with shape `(batch, seq_len_kv, num_heads_kv, head_size)`.
///
/// The resulting tensor has dimensions `(batch, seq_len_q, num_heads_q, head_size)`.
pub fn flash_attn(

View on GitHub (pinned to d5fee525bf)

Solutions

1. Convert q/k/v to DType::F16 or DType::BF16 with .to_dtype() before the call
2. Load the model in bf16/f16 (e.g. dtype from config) instead of f32
3. Fall back to standard attention (candle_nn::Sdpa) when running in f32

Example fix

// before
let (q, k, v) = (q.to_device(&dev)?, k.to_device(&dev)?, v.to_device(&dev)?);
flash_attn(&q, &k, &v, None, scale, true)?
// after
let dt = q.dtype();
let (q, k, v) = (
    q.to_dtype(DType::BF16)?.to_device(&dev)?,
    k.to_dtype(DType::BF16)?.to_device(&dev)?,
    v.to_dtype(DType::BF16)?.to_device(&dev)?,
);
flash_attn(&q, &k, &v, None, scale, true)?

fn ensure_half(t: &Tensor, dev: &Device) -> candle::Result<Tensor> {
    match t.dtype() {
        candle::DType::F16 | candle::DType::BF16 => Ok(t.clone()),
        _ => t.to_dtype(candle::DType::BF16)?.to_device(dev),
    }
}

fn is_half(t: &Tensor) -> bool {
    matches!(t.dtype(), candle::DType::F16 | candle::DType::BF16)
}

match flash_attn(&q, &k, &v, None, scale, causal) {
    Err(e) if e.to_string().contains("only supported for f16/bf16") => {
        let (q, k, v) = (q.to_dtype(DType::BF16)?, k.to_dtype(DType::BF16)?, v.to_dtype(DType::BF16)?);
        flash_attn(&q, &k, &v, None, scale, causal)?
    }
    r => r?,
}

When it happens

Trigger: Calling flash_attn / FlashAttnV2 with q (and by contract k/v) tensors of dtype f32 — typical when the model runs in full float32 precision, or when tensors were upcast by prior ops.

Common situations: Running a model with dtype f32 weights, forgetting to call .to_dtype(DType::BF16/F16) after loading f32 safetensors, converting a model port that keeps hidden states in f32 while using flash-attn layers.

Related errors

• only f32 can be quantized
• seqlens_q has to be contiguous
• seqlens_k must be a cuda tensor
• seqlens_k has to be contiguous
• shape mismatch alibi_slopes {:?}, expected {:?}

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/29dda99edce26720. Report an issue: GitHub.