{"record":{"id":"cbcfce4764f817ac","repo":"huggingface/candle","slug":"block-table-must-be-a-cuda-tensor","errorCode":null,"errorMessage":"block_table must be a cuda tensor","messagePattern":"block_table must be a cuda tensor","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-flash-attn/src/lib.rs","lineNumber":505,"sourceCode":"            Some((o1, o2)) => seqlens_q.slice(o1..o2),\n            None => candle::bail!(\"seqlens_q has to be contiguous\"),\n        };\n\n        let (seqlens_k, seqlens_k_layout) = self.seqlens_k.storage_and_layout();\n        let seqlens_k = match &*seqlens_k {\n            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!\n            _ => candle::bail!(\"seqlens_k must be a cuda tensor\"),\n        };\n        let seqlens_k = match seqlens_k_layout.contiguous_offsets() {\n            Some((o1, o2)) => seqlens_k.slice(o1..o2),\n            None => candle::bail!(\"seqlens_k has to be contiguous\"),\n        };\n\n        let block_table = if let Some(block_table) = self.block_table.as_ref() {\n            let (block_table_storage, block_table_layout) = block_table.storage_and_layout();\n            match &*block_table_storage {\n                candle::Storage::Cuda(_) => {}\n                _ => candle::bail!(\"block_table must be a cuda tensor\"),\n            }\n            let block_table_stride = block_table_layout.shape().dims2()?.1;\n            if block_table_layout.stride().last().copied() != Some(1) {\n                candle::bail!(\"block_table last dimension must be contiguous\")\n            }\n            Some((\n                block_table_storage,\n                block_table_layout.start_offset(),\n                block_table_stride,\n            ))\n        } else {\n            None\n        };\n\n        let q = q.as_cuda_slice::<T>()?;\n        let k = k.as_cuda_slice::<T>()?;\n        let v = v.as_cuda_slice::<T>()?;\n        let q = q.slice(q_l.start_offset()..);","sourceCodeStart":487,"sourceCodeEnd":523,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-flash-attn/src/lib.rs#L487-L523","documentation":"Raised during paged-attention flash-attn forward in candle-flash-attn/src/lib.rs when an optional `block_table` is supplied but its storage is not CUDA memory. The block table (2D mapping of logical to physical KV blocks) must reside on the CUDA device to be passed to the kernel.","triggerScenarios":"Calling flash_attn_varlen with Some(block_table) whose tensor is on CPU (or another backend), e.g. a page-table built host-side and never transferred.","commonSituations":"Paged-KV cache bookkeeping (block tables) computed on CPU per decode step and passed directly; loading tables from checkpoint data on the wrong device.","solutions":["Move block_table to the CUDA device with .to_device(&dev) before the call","Create/update the block table directly on GPU","If not using paged attention, pass None for block_table"],"exampleFix":"// before\nlet block_table = Tensor::from_vec(pages, (batch, max_blocks), &Device::Cpu)?;\nflash_attn_varlen(&q, &k, &v, &sq, &sk, softmax_scale, max_q, max_k, None, window, Some(block_table))?\n// after\nlet block_table = Tensor::from_vec(pages, (batch, max_blocks), &Device::Cpu)?.to_device(&dev)?;\nflash_attn_varlen(&q, &k, &v, &sq, &sk, softmax_scale, max_q, max_k, None, window, Some(block_table))?","handlingStrategy":"validation","validationCode":"fn ensure_block_table_cuda(bt: &Tensor) -> candle::Result<Tensor> {\n    if bt.dims().len() != 2 { candle::bail!(\"block_table must be 2D\"); }\n    if bt.device().is_cuda() { Ok(bt.clone()) } else { bt.to_device(&Device::new_cuda(0)?) }\n}","typeGuard":"fn block_table_ready(t: &Tensor) -> bool { t.dims().len() == 2 && t.device().is_cuda() }","tryCatchPattern":"let block_table = block_table.map(|bt| bt.to_device(q.device())).transpose()?;\nmatch flash_attn_varlen(&q, &k, &v, &sq, &sk, scale, max_q, max_k, None, None, block_table.as_ref()) {\n    Err(e) if e.to_string().contains(\"block_table must be a cuda tensor\") => { /* fix device, retry */ }\n    r => r?,\n}","preventionTips":["Upload block tables to GPU when the paged KV cache is created","Keep all paged-attention bookkeeping tensors on one device","Pass None for block_table when paged attention is unused"],"tags":["cuda","flash-attention","device-mismatch","paged-attention"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}