huggingface/candle · error

flash-attn-varlen expects input tensors of rank 3 (q: {q_ran

Error message

flash-attn-varlen expects input tensors of rank 3 (q: {q_rank}, k: {k_rank}, v: {v_rank}

What it means

This error comes from the varlen CUDA flash-attention kernel wrapper in candle-flash-attn. When no paged attention (block_table) is used, the kernel requires q, k, and v to be rank-3 tensors shaped (total_q, num_heads, head_dim). The wrapper checks q_stride.len(), k_stride.len(), v_stride.len() and bails if any non-paged tensor's rank differs from 3.

Source

Thrown at candle-flash-attn/src/lib.rs:539

        let k = k.as_cuda_slice::<T>()?;
        let v = v.as_cuda_slice::<T>()?;
        let q = q.slice(q_l.start_offset()..);
        let k = k.slice(k_l.start_offset()..);
        let v = v.slice(v_l.start_offset()..);

        let q_stride = q_l.stride();
        let k_stride = k_l.stride();
        let v_stride = v_l.stride();
        let o_stride = out_l.stride();

        let q_rank = q_stride.len();
        let k_rank = k_stride.len();
        let v_rank = v_stride.len();
        let o_rank = o_stride.len();

        let paged = block_table.is_some();
        if q_rank != 3 || (!paged && k_rank != 3) || (!paged && v_rank != 3) {
            candle::bail!(
                "flash-attn-varlen expects input tensors of rank 3 (q: {q_rank}, k: {k_rank}, v: {v_rank}"
            )
        }
        if paged && (k_rank != 4 || v_rank != 4) {
            candle::bail!(
                "flash-attn-varlen paged expects k/v tensors of rank 4 (k: {k_rank}, v: {v_rank})"
            )
        }
        if q_stride[q_rank - 1] != 1 {
            candle::bail!("the last dim of q must be contiguous {q_stride:?}")
        }
        if k_stride[k_rank - 1] != 1 {
            candle::bail!("the last dim of k must be contiguous {k_stride:?}")
        }
        if v_stride[v_rank - 1] != 1 {
            candle::bail!("the last dim of v must be contiguous {v_stride:?}")
        }

View on GitHub (pinned to d5fee525bf)

Solutions

1. Flatten the batch/sequence dims of q into (total_q, num_heads, head_dim), e.g. q.reshape((total_q, H, D))
2. For non-paged calls, reshape k and v to rank 3 (total_k, num_heads_k, head_dim) to match q
3. If you actually want batched paged attention, provide a block_table (paged mode) so k/v may be rank 4
4. If you want padded batch attention, use FlashAttention (non-varlen) instead of FlashAttentionVarLen

Example fix

// before
let q = q.reshape((b, s * q_len, h, d))?; // rank 4
let out = attn_fwd.forward(&q, &k, &v, ...)?;
// after
let q = q.reshape((b * s, h, d))?; // rank 3: (total_q, heads, head_dim)
let k = k.reshape((total_k, h_kv, d))?;
let v = v.reshape((total_k, h_kv, d))?;
let out = attn_fwd.forward(&q, &k, &v, ...)?;

fn check_rank3(t: &candle_core::Tensor, name: &str) -> candle_core::Result<()> {
    let d = t.dims();
    if d.len() != 3 {
        candle_core::bail!("{name} must be rank 3 (total, heads, head_dim), got {:?}", d);
    }
    Ok(())
}
// call before forward (non-paged):
check_rank3(&q, "q")?; check_rank3(&k, "k")?; check_rank3(&v, "v")?;

fn is_rank3(t: &candle_core::Tensor) -> bool { t.dims().len() == 3 }

match attn.forward(&q, &k, &v, &seqlens_q, &seqlens_k, None) {
    Ok(out) => out,
    Err(e) if e.to_string().contains("rank 3") => {
        let out = attn.forward(&q.reshape((total_q, h, d))?, &k, &v, &seqlens_q, &seqlens_k, None)?;
        out
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling FlashAttentionVarLen::forward (cuda_fwd_t) with a q tensor not of shape (total_q, H, D), or with k or v not rank 3 when no block_table was configured (e.g. tensors still carrying a batch dimension: (B, S, H, D) rank 4).

Common situations: Passing batched (B, S, H, D) tensors directly instead of flattening to packed varlen layout; forgetting to squeeze/reshape after an attention refactor; calling the varlen API when the non-paged flash-attn API was intended; upgrading candle and switching from padded batch attention to varlen without reshaping inputs.

Related errors

• shape mismatch alibi_slopes {:?}, expected {:?}
• flash-attn-varlen paged expects k/v tensors of rank 4 (k: {k
• seqlens_q has to be contiguous
• seqlens_k must be a cuda tensor
• seqlens_k has to be contiguous

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/455f6f90bb5892e8. Report an issue: GitHub.