huggingface/candle · error

block_table must be a cuda tensor

Error message

block_table must be a cuda tensor

What it means

Raised during paged-attention flash-attn forward in candle-flash-attn/src/lib.rs when an optional `block_table` is supplied but its storage is not CUDA memory. The block table (2D mapping of logical to physical KV blocks) must reside on the CUDA device to be passed to the kernel.

Source

Thrown at candle-flash-attn/src/lib.rs:505

            Some((o1, o2)) => seqlens_q.slice(o1..o2),
            None => candle::bail!("seqlens_q has to be contiguous"),
        };

        let (seqlens_k, seqlens_k_layout) = self.seqlens_k.storage_and_layout();
        let seqlens_k = match &*seqlens_k {
            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!
            _ => candle::bail!("seqlens_k must be a cuda tensor"),
        };
        let seqlens_k = match seqlens_k_layout.contiguous_offsets() {
            Some((o1, o2)) => seqlens_k.slice(o1..o2),
            None => candle::bail!("seqlens_k has to be contiguous"),
        };

        let block_table = if let Some(block_table) = self.block_table.as_ref() {
            let (block_table_storage, block_table_layout) = block_table.storage_and_layout();
            match &*block_table_storage {
                candle::Storage::Cuda(_) => {}
                _ => candle::bail!("block_table must be a cuda tensor"),
            }
            let block_table_stride = block_table_layout.shape().dims2()?.1;
            if block_table_layout.stride().last().copied() != Some(1) {
                candle::bail!("block_table last dimension must be contiguous")
            }
            Some((
                block_table_storage,
                block_table_layout.start_offset(),
                block_table_stride,
            ))
        } else {
            None
        };

        let q = q.as_cuda_slice::<T>()?;
        let k = k.as_cuda_slice::<T>()?;
        let v = v.as_cuda_slice::<T>()?;
        let q = q.slice(q_l.start_offset()..);

View on GitHub (pinned to d5fee525bf)

Solutions

1. Move block_table to the CUDA device with .to_device(&dev) before the call
2. Create/update the block table directly on GPU
3. If not using paged attention, pass None for block_table

Example fix

// before
let block_table = Tensor::from_vec(pages, (batch, max_blocks), &Device::Cpu)?;
flash_attn_varlen(&q, &k, &v, &sq, &sk, softmax_scale, max_q, max_k, None, window, Some(block_table))?
// after
let block_table = Tensor::from_vec(pages, (batch, max_blocks), &Device::Cpu)?.to_device(&dev)?;
flash_attn_varlen(&q, &k, &v, &sq, &sk, softmax_scale, max_q, max_k, None, window, Some(block_table))?

fn ensure_block_table_cuda(bt: &Tensor) -> candle::Result<Tensor> {
    if bt.dims().len() != 2 { candle::bail!("block_table must be 2D"); }
    if bt.device().is_cuda() { Ok(bt.clone()) } else { bt.to_device(&Device::new_cuda(0)?) }
}

fn block_table_ready(t: &Tensor) -> bool { t.dims().len() == 2 && t.device().is_cuda() }

let block_table = block_table.map(|bt| bt.to_device(q.device())).transpose()?;
match flash_attn_varlen(&q, &k, &v, &sq, &sk, scale, max_q, max_k, None, None, block_table.as_ref()) {
    Err(e) if e.to_string().contains("block_table must be a cuda tensor") => { /* fix device, retry */ }
    r => r?,
}

When it happens

Trigger: Calling flash_attn_varlen with Some(block_table) whose tensor is on CPU (or another backend), e.g. a page-table built host-side and never transferred.

Common situations: Paged-KV cache bookkeeping (block tables) computed on CPU per decode step and passed directly; loading tables from checkpoint data on the wrong device.

Related errors

• seqlens_k must be a cuda tensor
• alibi_slopes must be a cuda tensor
• seqlens_q must be a cuda tensor
• seqlens_k must be a cuda tensor
• block_table last dimension must be contiguous

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/cbcfce4764f817ac. Report an issue: GitHub.