huggingface/candle · error

seqlens_q has to be contiguous

Error message

seqlens_q has to be contiguous

What it means

The varlen forward requires seqlens_q (cu_seqlens_q, cumulative query-sequence offsets) to occupy a single contiguous memory range, because it is converted into a raw CUDA slice via contiguous_offsets(). The check returned None, so the tensor's layout is strided or a view not expressible as one contiguous run. The library bails rather than silently misreading offsets on the GPU.

Source

Thrown at candle-flash-attn-v3/src/lib.rs:468

        k: &candle::CudaStorage,
        k_l: &Layout,
        v: &candle::CudaStorage,
        v_l: &Layout,
        is_bf16: bool,
    ) -> Result<(candle::CudaStorage, Shape)> {
        // https://github.com/Dao-AILab/flash-attention/blob/0dfb28174333d9eefb7c1dd4292690a8458d1e89/hopper/flash_api.cpp
        let dev = q.device();
        let out_shape = q_l.shape().clone();
        let out_l = Layout::contiguous(&out_shape);

        let (seqlens_q, seqlens_q_layout) = self.seqlens_q.storage_and_layout();
        let seqlens_q = match &*seqlens_q {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_q must be a cuda tensor"),
        };
        let seqlens_q = match seqlens_q_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_q.slice(o1..o2),
            None => candle::bail!("seqlens_q has to be contiguous"),
        };

        let (seqlens_k, seqlens_k_layout) = self.seqlens_k.storage_and_layout();
        let seqlens_k = match &*seqlens_k {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_k must be a cuda tensor"),
        };
        let seqlens_k = match seqlens_k_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_k.slice(o1..o2),
            None => candle::bail!("seqlens_k has to be contiguous"),
        };

        let q = q.as_cuda_slice::<T>()?;
        let k = k.as_cuda_slice::<T>()?;
        let v = v.as_cuda_slice::<T>()?;
        let q = q.slice(q_l.start_offset()..);
        let k = k.slice(k_l.start_offset()..);
        let v = v.slice(v_l.start_offset()..);

View on GitHub (pinned to d5fee525bf)

Solutions

1. Call .contiguous() on seqlens_q before passing it to the forward call.
2. Create seqlens_q fresh as a 1-D contiguous u32 CUDA tensor (Tensor::from_vec on the device) instead of slicing.
3. Insert .contiguous() at the pipeline boundary where offsets are handed to the attention module.

Example fix

// before
let seqlens_q = offsets.i(1..)?; // strided view
flash_attn_varlen_forward(&q, &k, &v, &seqlens_q, &seqlens_k, ...)?;
// after
let seqlens_q = offsets.i(1..)?.to_device(q.device())?.contiguous()?;
flash_attn_varlen_forward(&q, &k, &v, &seqlens_q, &seqlens_k, ...)?;

fn check_seqlens_q(t: &candle_core::Tensor) -> candle_core::Result<()> {
    if t.device().is_cpu() { candle_core::bail!("seqlens_q must be on CUDA"); }
    if t.dtype() != candle_core::DType::U32 { candle_core::bail!("seqlens_q must be u32"); }
    if t.contiguous_offsets().is_none() { candle_core::bail!("seqlens_q not contiguous; call .contiguous()"); }
    Ok(())
}
// run before the forward call: check_seqlens_q(&seqlens_q)?;

fn is_contiguous_cuda(t: &candle_core::Tensor) -> bool {
    !t.device().is_cpu() && t.contiguous_offsets().is_some()
}

match flash_attn_varlen_forward(&q, &k, &v, &seqlens_q, &seqlens_k, ...) {
    Ok(out) => out,
    Err(e) if e.to_string().contains("seqlens_q has to be contiguous") => {
        let sq = seqlens_q.contiguous()?;
        flash_attn_varlen_forward(&q, &k, &v, &sq, &seqlens_k, ...)?
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling the varlen forward (cuda_fwd_t path) with seqlens_q produced by a strided slice of a larger tensor, a transpose/permute view, or chained narrow/index_select ops leaving a non-contiguous layout.

Common situations: Slicing cu_seqlens out of a combined offsets buffer; building offsets on CPU and moving only a sub-view to CUDA; reusing metadata tensors reshaped for other kernels without materializing a copy.

Related errors

• seqlens_k has to be contiguous
• seqlens_q has to be contiguous
• seqlens_k has to be contiguous
• block_table last dimension must be contiguous
• input has to be contiguous

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/e7d305f77aed5235. Report an issue: GitHub.