{"record":{"id":"f3e0ef4a3efc23b1","repo":"huggingface/candle","slug":"seqlens-q-must-be-a-cuda-tensor-f3e0ef","errorCode":null,"errorMessage":"seqlens_q must be a cuda tensor","messagePattern":"seqlens_q must be a cuda tensor","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-flash-attn/src/lib.rs","lineNumber":484,"sourceCode":"    >(\n        &self,\n        q: &candle::CudaStorage,\n        q_l: &Layout,\n        k: &candle::CudaStorage,\n        k_l: &Layout,\n        v: &candle::CudaStorage,\n        v_l: &Layout,\n        is_bf16: bool,\n    ) -> Result<(candle::CudaStorage, Shape)> {\n        // https://github.com/Dao-AILab/flash-attention/blob/184b992dcb2a0890adaa19eb9b541c3e4f9d2a08/csrc/flash_attn/flash_api.cpp#L327\n        let dev = q.device();\n        let out_shape = q_l.shape().clone();\n        let out_l = Layout::contiguous(&out_shape);\n\n        let (seqlens_q, seqlens_q_layout) = self.seqlens_q.storage_and_layout();\n        let seqlens_q = match &*seqlens_q {\n            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!\n            _ => candle::bail!(\"seqlens_q must be a cuda tensor\"),\n        };\n        let seqlens_q = match seqlens_q_layout.contiguous_offsets() {\n            Some((o1, o2)) => seqlens_q.slice(o1..o2),\n            None => candle::bail!(\"seqlens_q has to be contiguous\"),\n        };\n\n        let (seqlens_k, seqlens_k_layout) = self.seqlens_k.storage_and_layout();\n        let seqlens_k = match &*seqlens_k {\n            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!\n            _ => candle::bail!(\"seqlens_k must be a cuda tensor\"),\n        };\n        let seqlens_k = match seqlens_k_layout.contiguous_offsets() {\n            Some((o1, o2)) => seqlens_k.slice(o1..o2),\n            None => candle::bail!(\"seqlens_k has to be contiguous\"),\n        };\n\n        let block_table = if let Some(block_table) = self.block_table.as_ref() {\n            let (block_table_storage, block_table_layout) = block_table.storage_and_layout();","sourceCodeStart":466,"sourceCodeEnd":502,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-flash-attn/src/lib.rs#L466-L502","documentation":"In the varlen (paged/ALiBi-v2) flash-attn path, the seqlens_q tensor (per-batch cumulative query sequence lengths, cu_seqlens) must live on the CUDA device as u32 storage. Non-CUDA storage triggers this bail.","triggerScenarios":"Calling flash_attn_varlen (or FlashAttnV2 with seqlens set) while the seqlens_q tensor remains on CPU; creating seqlens with Device::Cpu and only moving q/k/v to GPU.","commonSituations":"Building cu_seqlens on CPU from Python-side/tokenizer batch metadata and forgetting .to_device, mixed-device model setups, deserializing seqlens from safetensors onto the wrong device.","solutions":["Move seqlens_q to the CUDA device with .to_device(&dev) before calling","Ensure it is u32 typed on CUDA (as_cuda_slice::<u32>() also enforces dtype)","Create the tensor directly on the CUDA device when constructing batch metadata"],"exampleFix":"// before\nlet seqlens_q = Tensor::from_vec(seqlens, (batch + 1,), &Device::Cpu)?;\nflash_attn_varlen(&q, &k, &v, &seqlens_q, &seqlens_k, ..)?\n// after\nlet seqlens_q = Tensor::from_vec(seqlens, (batch + 1,), &Device::Cpu)?.to_device(&dev)?;\nflash_attn_varlen(&q, &k, &v, &seqlens_q, &seqlens_k, ..)?","handlingStrategy":"validation","validationCode":"fn ensure_u32_cuda(t: &Tensor) -> candle::Result<Tensor> {\n    if t.dtype() != candle::DType::U32 { candle::bail!(\"seqlens_q must be u32\"); }\n    if t.device().is_cuda() { Ok(t.clone()) } else { t.to_device(&Device::new_cuda(0)?) }\n}","typeGuard":"fn seqlens_ready(t: &Tensor) -> bool { t.dtype() == candle::DType::U32 && t.device().is_cuda() }","tryCatchPattern":"let seqlens_q = seqlens_q.to_device(k.device())?;\nmatch flash_attn_varlen(&q, &k, &v, &seqlens_q, &seqlens_k, scale, max_q, max_k, None, None, None) {\n    Err(e) if e.to_string().contains(\"seqlens_q must be a cuda tensor\") => { /* fix device, retry */ }\n    r => r?,\n}","preventionTips":["Transfer every varlen input (including seqlens) with .to_device at batch prep time","Keep seqlens dtype u32 as the API expects","Build batch metadata tensors on the same device as q/k/v"],"tags":["cuda","flash-attention","device-mismatch","varlen"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}