{"record":{"id":"036ccc409ccd2e22","repo":"huggingface/candle","slug":"seqlens-k-must-be-a-cuda-tensor","errorCode":null,"errorMessage":"seqlens_k must be a cuda tensor","messagePattern":"seqlens_k must be a cuda tensor","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-flash-attn-v3/src/lib.rs","lineNumber":474,"sourceCode":"        // https://github.com/Dao-AILab/flash-attention/blob/0dfb28174333d9eefb7c1dd4292690a8458d1e89/hopper/flash_api.cpp\n        let dev = q.device();\n        let out_shape = q_l.shape().clone();\n        let out_l = Layout::contiguous(&out_shape);\n\n        let (seqlens_q, seqlens_q_layout) = self.seqlens_q.storage_and_layout();\n        let seqlens_q = match &*seqlens_q {\n            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!\n            _ => candle::bail!(\"seqlens_q must be a cuda tensor\"),\n        };\n        let seqlens_q = match seqlens_q_layout.contiguous_offsets() {\n            Some((o1, o2)) => seqlens_q.slice(o1..o2),\n            None => candle::bail!(\"seqlens_q has to be contiguous\"),\n        };\n\n        let (seqlens_k, seqlens_k_layout) = self.seqlens_k.storage_and_layout();\n        let seqlens_k = match &*seqlens_k {\n            candle::Storage::Cuda(c) => c.as_cuda_slice::<u32>()?, // Should be i32!\n            _ => candle::bail!(\"seqlens_k must be a cuda tensor\"),\n        };\n        let seqlens_k = match seqlens_k_layout.contiguous_offsets() {\n            Some((o1, o2)) => seqlens_k.slice(o1..o2),\n            None => candle::bail!(\"seqlens_k has to be contiguous\"),\n        };\n\n        let q = q.as_cuda_slice::<T>()?;\n        let k = k.as_cuda_slice::<T>()?;\n        let v = v.as_cuda_slice::<T>()?;\n        let q = q.slice(q_l.start_offset()..);\n        let k = k.slice(k_l.start_offset()..);\n        let v = v.slice(v_l.start_offset()..);\n\n        let q_stride = q_l.stride();\n        let k_stride = k_l.stride();\n        let v_stride = v_l.stride();\n        let o_stride = out_l.stride();\n","sourceCodeStart":456,"sourceCodeEnd":492,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-flash-attn-v3/src/lib.rs#L456-L492","documentation":"The varlen kernel reads cu_seqlens_k (cumulative key-sequence offsets) directly on the GPU, so seqlens_k must live in candle::Storage::Cuda. storage_and_layout() returned non-CUDA storage, meaning the tensor is on CPU or another backend. The source also notes the slice is read as u32 though i32 offsets are the natural dtype.","triggerScenarios":"Calling the varlen forward with seqlens_k still on the CPU device or on a different device than q/k/v — e.g. created with Tensor::from_vec on Device::Cpu and never moved.","commonSituations":"Moving model weights to CUDA but forgetting metadata tensors; q/k/v moved via .to_device while seqlens were left behind; constructing offsets before the target device is known.","solutions":["Move seqlens_k to the same CUDA device as q: seqlens_k.to_device(q.device())?.","Create cu_seqlens_k directly on the CUDA device via Tensor::from_vec(vec, (n + 1,), device).","Add a device-equality assertion for all forward inputs at the attention module boundary."],"exampleFix":"// before\nlet seqlens_k = Tensor::from_vec(k_offsets, (n + 1,), &Device::Cpu)?;\n// after\nlet seqlens_k = Tensor::from_vec(k_offsets, (n + 1,), q.device())?;","handlingStrategy":"validation","validationCode":"if seqlens_k.device().is_cpu() {\n    let seqlens_k = seqlens_k.to_device(q.device())?;\n}\ndebug_assert!(!seqlens_k.device().is_cpu(), \"seqlens_k must be on the CUDA device\");","typeGuard":"fn is_on_cuda(t: &candle_core::Tensor) -> bool { !t.device().is_cpu() }","tryCatchPattern":"match forward(&q, &k, &v, &seqlens_q, &seqlens_k, ...) {\n    Err(e) if e.to_string().contains(\"seqlens_k must be a cuda tensor\") => {\n        let sk = seqlens_k.to_device(q.device())?;\n        forward(&q, &k, &v, &seqlens_q, &sk, ...)\n    }\n    other => other,\n}","preventionTips":["Create seqlens tensors on the same device as q/k/v from the start (pass q.device() into Tensor::from_vec).","When moving a model to CUDA, move metadata tensors (cu_seqlens) too, not just weights.","Assert device equality for all forward inputs at the module boundary."],"tags":["cuda","device-mismatch","flash-attention","tensor-placement"],"backgroundTag":"cuda-device-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}