huggingface/candle · error

seqlens_q must be a cuda tensor

Error message

seqlens_q must be a cuda tensor

What it means

In the varlen (paged/ALiBi-v2) flash-attn path, the seqlens_q tensor (per-batch cumulative query sequence lengths, cu_seqlens) must live on the CUDA device as u32 storage. Non-CUDA storage triggers this bail.

Source

Thrown at candle-flash-attn/src/lib.rs:484

    >(
        &self,
        q: &candle::CudaStorage,
        q_l: &Layout,
        k: &candle::CudaStorage,
        k_l: &Layout,
        v: &candle::CudaStorage,
        v_l: &Layout,
        is_bf16: bool,
    ) -> Result<(candle::CudaStorage, Shape)> {
        // https://github.com/Dao-AILab/flash-attention/blob/184b992dcb2a0890adaa19eb9b541c3e4f9d2a08/csrc/flash_attn/flash_api.cpp#L327
        let dev = q.device();
        let out_shape = q_l.shape().clone();
        let out_l = Layout::contiguous(&out_shape);

        let (seqlens_q, seqlens_q_layout) = self.seqlens_q.storage_and_layout();
        let seqlens_q = match &*seqlens_q {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_q must be a cuda tensor"),
        };
        let seqlens_q = match seqlens_q_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_q.slice(o1..o2),
            None => candle::bail!("seqlens_q has to be contiguous"),
        };

        let (seqlens_k, seqlens_k_layout) = self.seqlens_k.storage_and_layout();
        let seqlens_k = match &*seqlens_k {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_k must be a cuda tensor"),
        };
        let seqlens_k = match seqlens_k_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_k.slice(o1..o2),
            None => candle::bail!("seqlens_k has to be contiguous"),
        };

        let block_table = if let Some(block_table) = self.block_table.as_ref() {
            let (block_table_storage, block_table_layout) = block_table.storage_and_layout();

View on GitHub (pinned to d5fee525bf)

Solutions

1. Move seqlens_q to the CUDA device with .to_device(&dev) before calling
2. Ensure it is u32 typed on CUDA (as_cuda_slice::<u32>() also enforces dtype)
3. Create the tensor directly on the CUDA device when constructing batch metadata

Example fix

// before
let seqlens_q = Tensor::from_vec(seqlens, (batch + 1,), &Device::Cpu)?;
flash_attn_varlen(&q, &k, &v, &seqlens_q, &seqlens_k, ..)?
// after
let seqlens_q = Tensor::from_vec(seqlens, (batch + 1,), &Device::Cpu)?.to_device(&dev)?;
flash_attn_varlen(&q, &k, &v, &seqlens_q, &seqlens_k, ..)?

fn ensure_u32_cuda(t: &Tensor) -> candle::Result<Tensor> {
    if t.dtype() != candle::DType::U32 { candle::bail!("seqlens_q must be u32"); }
    if t.device().is_cuda() { Ok(t.clone()) } else { t.to_device(&Device::new_cuda(0)?) }
}

fn seqlens_ready(t: &Tensor) -> bool { t.dtype() == candle::DType::U32 && t.device().is_cuda() }

let seqlens_q = seqlens_q.to_device(k.device())?;
match flash_attn_varlen(&q, &k, &v, &seqlens_q, &seqlens_k, scale, max_q, max_k, None, None, None) {
    Err(e) if e.to_string().contains("seqlens_q must be a cuda tensor") => { /* fix device, retry */ }
    r => r?,
}

When it happens

Trigger: Calling flash_attn_varlen (or FlashAttnV2 with seqlens set) while the seqlens_q tensor remains on CPU; creating seqlens with Device::Cpu and only moving q/k/v to GPU.

Common situations: Building cu_seqlens on CPU from Python-side/tokenizer batch metadata and forgetting .to_device, mixed-device model setups, deserializing seqlens from safetensors onto the wrong device.

Related errors

• seqlens_k must be a cuda tensor
• seqlens_k must be a cuda tensor
• alibi_slopes must be a cuda tensor
• seqlens_q has to be contiguous
• seqlens_k has to be contiguous

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/f3e0ef4a3efc23b1. Report an issue: GitHub.