huggingface/candle · error

seqlens_k must be a cuda tensor

Error message

seqlens_k must be a cuda tensor

What it means

The varlen kernel reads cu_seqlens_k (cumulative key-sequence offsets) directly on the GPU, so seqlens_k must live in candle::Storage::Cuda. storage_and_layout() returned non-CUDA storage, meaning the tensor is on CPU or another backend. The source also notes the slice is read as u32 though i32 offsets are the natural dtype.

Solutions

1. Move seqlens_k to the same CUDA device as q: seqlens_k.to_device(q.device())?.
2. Create cu_seqlens_k directly on the CUDA device via Tensor::from_vec(vec, (n + 1,), device).
3. Add a device-equality assertion for all forward inputs at the attention module boundary.

Example fix

// before
let seqlens_k = Tensor::from_vec(k_offsets, (n + 1,), &Device::Cpu)?;
// after
let seqlens_k = Tensor::from_vec(k_offsets, (n + 1,), q.device())?;

if seqlens_k.device().is_cpu() {
    let seqlens_k = seqlens_k.to_device(q.device())?;
}
debug_assert!(!seqlens_k.device().is_cpu(), "seqlens_k must be on the CUDA device");

fn is_on_cuda(t: &candle_core::Tensor) -> bool { !t.device().is_cpu() }

match forward(&q, &k, &v, &seqlens_q, &seqlens_k, ...) {
    Err(e) if e.to_string().contains("seqlens_k must be a cuda tensor") => {
        let sk = seqlens_k.to_device(q.device())?;
        forward(&q, &k, &v, &seqlens_q, &sk, ...)
    }
    other => other,
}

When it happens

Trigger: Calling the varlen forward with seqlens_k still on the CPU device or on a different device than q/k/v — e.g. created with Tensor::from_vec on Device::Cpu and never moved.

Common situations: Moving model weights to CUDA but forgetting metadata tensors; q/k/v moved via .to_device while seqlens were left behind; constructing offsets before the target device is known.

Related errors

• alibi_slopes must be a cuda tensor
• block_table must be a cuda tensor
• seqlens_k must be a cuda tensor
• seqlens_q must be a cuda tensor
• block_table last dimension must be contiguous

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/036ccc409ccd2e22. Report an issue: GitHub.

Appendix: source

Thrown at candle-flash-attn-v3/src/lib.rs:474

        // https://github.com/Dao-AILab/flash-attention/blob/0dfb28174333d9eefb7c1dd4292690a8458d1e89/hopper/flash_api.cpp
        let dev = q.device();
        let out_shape = q_l.shape().clone();
        let out_l = Layout::contiguous(&out_shape);

        let (seqlens_q, seqlens_q_layout) = self.seqlens_q.storage_and_layout();
        let seqlens_q = match &*seqlens_q {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_q must be a cuda tensor"),
        };
        let seqlens_q = match seqlens_q_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_q.slice(o1..o2),
            None => candle::bail!("seqlens_q has to be contiguous"),
        };

        let (seqlens_k, seqlens_k_layout) = self.seqlens_k.storage_and_layout();
        let seqlens_k = match &*seqlens_k {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_k must be a cuda tensor"),
        };
        let seqlens_k = match seqlens_k_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_k.slice(o1..o2),
            None => candle::bail!("seqlens_k has to be contiguous"),
        };

        let q = q.as_cuda_slice::<T>()?;
        let k = k.as_cuda_slice::<T>()?;
        let v = v.as_cuda_slice::<T>()?;
        let q = q.slice(q_l.start_offset()..);
        let k = k.slice(k_l.start_offset()..);
        let v = v.slice(v_l.start_offset()..);

        let q_stride = q_l.stride();
        let k_stride = k_l.stride();
        let v_stride = v_l.stride();
        let o_stride = out_l.stride();

View on GitHub (pinned to d5fee525bf)