huggingface/candle · error

seqlens_k has to be contiguous

Error message

seqlens_k has to be contiguous

What it means

Raised in candle-flash-attn/src/lib.rs when the `seqlens_k` tensor's layout has no contiguous offsets, i.e. the tensor is not contiguous in memory (e.g. sliced or strided). The kernel reads it as a flat device slice, so call `.contiguous()` on it before invoking flash-attn.

Source

Thrown at candle-flash-attn/src/lib.rs:498

        let (seqlens_q, seqlens_q_layout) = self.seqlens_q.storage_and_layout();
        let seqlens_q = match &*seqlens_q {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_q must be a cuda tensor"),
        };
        let seqlens_q = match seqlens_q_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_q.slice(o1..o2),
            None => candle::bail!("seqlens_q has to be contiguous"),
        };

        let (seqlens_k, seqlens_k_layout) = self.seqlens_k.storage_and_layout();
        let seqlens_k = match &*seqlens_k {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_k must be a cuda tensor"),
        };
        let seqlens_k = match seqlens_k_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_k.slice(o1..o2),
            None => candle::bail!("seqlens_k has to be contiguous"),
        };

        let block_table = if let Some(block_table) = self.block_table.as_ref() {
            let (block_table_storage, block_table_layout) = block_table.storage_and_layout();
            match &*block_table_storage {
                candle::Storage::Cuda(_) => {}
                _ => candle::bail!("block_table must be a cuda tensor"),
            }
            let block_table_stride = block_table_layout.shape().dims2()?.1;
            if block_table_layout.stride().last().copied() != Some(1) {
                candle::bail!("block_table last dimension must be contiguous")
            }
            Some((
                block_table_storage,
                block_table_layout.start_offset(),
                block_table_stride,
            ))
        } else {

View on GitHub (pinned to d5fee525bf)

Solutions

1. Call .contiguous() on seqlens_k before the call
2. Store seqlens as a standalone flat contiguous tensor rather than a view
3. Rebuild the tensor with Tensor::from_vec when in doubt

Example fix

// before
let seqlens_k = big_buf.i((.., 0))?; // possibly non-contiguous
// after
let seqlens_k = big_buf.i((.., 0))?.contiguous()?;

fn ensure_contiguous_k(t: &Tensor) -> candle::Result<Tensor> {
    if t.is_contiguous() { Ok(t.clone()) } else { t.contiguous() }
}
let seqlens_k = ensure_contiguous_k(&seqlens_k)?;

fn contiguous_cuda_u32_k(t: &Tensor) -> bool {
    t.is_contiguous() && t.device().is_cuda() && t.dtype() == candle::DType::U32
}

match flash_attn_varlen(&q, &k, &v, &seqlens_q, &seqlens_k.contiguous()?, scale, max_q, max_k, None, None, None) {
    Err(e) if e.to_string().contains("has to be contiguous") => { /* rebuild seqlens_k */ }
    r => r?,
}

When it happens

Trigger: Passing a seqlens_k view created by narrow/slice/strided indexing without making it contiguous.

Common situations: Slicing cu_seqlens_k out of a larger pinned metadata buffer, transposing or reshaping metadata tensors before the attention call.

Related errors

• seqlens_q has to be contiguous
• seqlens_q has to be contiguous
• seqlens_k has to be contiguous
• seqlens_q must be a cuda tensor
• seqlens_k must be a cuda tensor

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/62cb8e0e8f9f0d24. Report an issue: GitHub.