huggingface/candle · error

seqlens_k has to be contiguous

Error message

seqlens_k has to be contiguous

What it means

Same contiguity precondition as seqlens_q, applied to seqlens_k: the tensor must map to a single contiguous offset range so it can be turned into a raw CUDA slice for the kernel. contiguous_offsets() returned None because the layout is strided or otherwise not contiguous.

Source

Thrown at candle-flash-attn-v3/src/lib.rs:478

        let (seqlens_q, seqlens_q_layout) = self.seqlens_q.storage_and_layout();
        let seqlens_q = match &*seqlens_q {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_q must be a cuda tensor"),
        };
        let seqlens_q = match seqlens_q_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_q.slice(o1..o2),
            None => candle::bail!("seqlens_q has to be contiguous"),
        };

        let (seqlens_k, seqlens_k_layout) = self.seqlens_k.storage_and_layout();
        let seqlens_k = match &*seqlens_k {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_k must be a cuda tensor"),
        };
        let seqlens_k = match seqlens_k_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_k.slice(o1..o2),
            None => candle::bail!("seqlens_k has to be contiguous"),
        };

        let q = q.as_cuda_slice::<T>()?;
        let k = k.as_cuda_slice::<T>()?;
        let v = v.as_cuda_slice::<T>()?;
        let q = q.slice(q_l.start_offset()..);
        let k = k.slice(k_l.start_offset()..);
        let v = v.slice(v_l.start_offset()..);

        let q_stride = q_l.stride();
        let k_stride = k_l.stride();
        let v_stride = v_l.stride();
        let o_stride = out_l.stride();

        let q_rank = q_stride.len();
        let k_rank = k_stride.len();
        let v_rank = v_stride.len();
        let o_rank = o_stride.len();

View on GitHub (pinned to d5fee525bf)

Solutions

1. Call .contiguous() on seqlens_k before the forward call.
2. Create seqlens_k as a fresh 1-D tensor on the CUDA device rather than a view of a larger buffer.
3. Materialize derived offsets, e.g. tensor.flatten_all()?.contiguous()?.

Example fix

// before
let seqlens_k = combined_offsets.i(..)?; // non-contiguous view
// after
let seqlens_k = combined_offsets.i(..)?.contiguous()?;

let seqlens_k = if seqlens_k.contiguous_offsets().is_none() {
    seqlens_k.contiguous()?
} else { seqlens_k };

fn is_contiguous(t: &candle_core::Tensor) -> bool { t.contiguous_offsets().is_some() }

match forward(&q, &k, &v, &seqlens_q, &seqlens_k, ...) {
    Err(e) if e.to_string().contains("seqlens_k has to be contiguous") => {
        let sk = seqlens_k.contiguous()?;
        forward(&q, &k, &v, &seqlens_q, &sk, ...)
    }
    other => other,
}

When it happens

Trigger: Passing a non-contiguous seqlens_k (strided slice, permuted view, or narrow of a larger offsets tensor) into the varlen forward's cuda_fwd_t path.

Common situations: Slicing cu_seqlens_k out of a shared offsets buffer; deriving offsets via ops that keep non-unit strides; reusing a transposed metadata tensor across kernels.

Related errors

• seqlens_q has to be contiguous
• seqlens_q has to be contiguous
• seqlens_k has to be contiguous
• block_table last dimension must be contiguous
• input has to be contiguous

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/e09cee8579ae3b53. Report an issue: GitHub.