{"record":{"id":"be733d054a5e97c1","repo":"huggingface/candle","slug":"alibi-slopes-must-be-a-cuda-tensor-be733d","errorCode":null,"errorMessage":"alibi_slopes must be a cuda tensor","messagePattern":"alibi_slopes must be a cuda tensor","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-flash-attn/src/lib.rs","lineNumber":112,"sourceCode":"                    \"DType mismatch alibi_slopes {:?}, expected {:?}\",\n                    alibi_slopes.dtype(),\n                    DType::F32\n                );\n            }\n\n            let (alibi_slopes, alibi_slopes_layout) = alibi_slopes.storage_and_layout();\n\n            if num_heads != alibi_slopes_layout.shape().dims1()? {\n                candle::bail!(\n                    \"shape mismatch alibi_slopes {:?}, expected {:?}\",\n                    alibi_slopes_layout.shape(),\n                    (num_heads)\n                );\n            }\n\n            let alibi_slopes = match &*alibi_slopes {\n                candle::Storage::Cuda(c) => c.as_cuda_slice::<f32>()?,\n                _ => candle::bail!(\"alibi_slopes must be a cuda tensor\"),\n            };\n\n            let alibi_slopes = alibi_slopes.slice(alibi_slopes_layout.start_offset()..);\n\n            // Dropping the guard here doesn't seem very safe.\n            let (ptr, _guard) = alibi_slopes.device_ptr(&stream);\n            ptr as *const core::ffi::c_void\n        } else {\n            std::ptr::null()\n        };\n\n        // if window_size_left > self.max_seqlen_k or None => -1\n        let mut window_size_left = self\n            .window_size_left\n            .filter(|v| v <= &seqlen_k)\n            .map(|v| v as i32)\n            .unwrap_or(-1);\n","sourceCodeStart":94,"sourceCodeEnd":130,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-flash-attn/src/lib.rs#L94-L130","documentation":"Raised by flash-attn's forward in candle-flash-attn/src/lib.rs when an optional `alibi_slopes` tensor is supplied but its underlying storage is not CUDA memory (e.g. it lives on CPU). The alibi slopes must reside on the same CUDA device as q/k/v to be passed as a raw device pointer to the kernel.","triggerScenarios":"Passing an alibi_slopes tensor that was created on / still resides on the CPU device while q/k/v are CUDA tensors; also occurs if the tensor's dtype is not f32, since as_cuda_slice::<f32>() would fail first, but a non-CUDA storage always triggers this bail.","commonSituations":"Creating slopes with Tensor::zeros on Device::Cpu by mistake, loading slopes from a checkpoint mapped to CPU and forgetting .to_device(&cuda_device), mixed-device pipelines after moving only q/k/v to GPU.","solutions":["Move the slopes tensor to the CUDA device with .to_device(&cuda_device) before the call","Ensure the whole model (including precomputed buffers like alibi slopes) is loaded on the same CUDA device","Check the tensor's device (t.device()) equals q.device() before calling"],"exampleFix":"// before\nlet slopes = Tensor::from_vec(vec![..], (num_heads,), &Device::Cpu)?;\n// after\nlet slopes = Tensor::from_vec(vec![..], (num_heads,), &Device::Cpu)?.to_device(&dev)?;","handlingStrategy":"validation","validationCode":"fn ensure_cuda(t: &Tensor) -> candle::Result<Tensor> {\n    if t.device().is_cuda() { Ok(t.clone()) } else { t.to_device(&Device::new_cuda(0)?) }\n}\nlet slopes = ensure_cuda(&slopes)?;","typeGuard":"fn is_cuda(t: &Tensor) -> bool { t.device().is_cuda() }","tryCatchPattern":"let slopes = slopes.to_device(q.device())?; // unify devices before call\nmatch flash_attn(&q, &k, &v, Some(&slopes), scale, causal) {\n    Err(e) if e.to_string().contains(\"must be a cuda tensor\") => { /* move tensors and retry once */ }\n    r => r?,\n}","preventionTips":["Create all auxiliary tensors directly on the model device","Call .to_device(q.device()) on every optional tensor argument","Check t.device() == q.device() in debug builds"],"tags":["cuda","flash-attention","device-mismatch","tensor-device"],"backgroundTag":"tensor-device-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}