huggingface/candle · error

alibi_slopes must be a cuda tensor

Error message

alibi_slopes must be a cuda tensor

What it means

Raised by flash-attn's forward in candle-flash-attn/src/lib.rs when an optional `alibi_slopes` tensor is supplied but its underlying storage is not CUDA memory (e.g. it lives on CPU). The alibi slopes must reside on the same CUDA device as q/k/v to be passed as a raw device pointer to the kernel.

Source

Thrown at candle-flash-attn/src/lib.rs:112

                    "DType mismatch alibi_slopes {:?}, expected {:?}",
                    alibi_slopes.dtype(),
                    DType::F32
                );
            }

            let (alibi_slopes, alibi_slopes_layout) = alibi_slopes.storage_and_layout();

            if num_heads != alibi_slopes_layout.shape().dims1()? {
                candle::bail!(
                    "shape mismatch alibi_slopes {:?}, expected {:?}",
                    alibi_slopes_layout.shape(),
                    (num_heads)
                );
            }

            let alibi_slopes = match &*alibi_slopes {
                candle::Storage::Cuda(c) => c.as_cuda_slice::<f32>()?,
                _ => candle::bail!("alibi_slopes must be a cuda tensor"),
            };

            let alibi_slopes = alibi_slopes.slice(alibi_slopes_layout.start_offset()..);

            // Dropping the guard here doesn't seem very safe.
            let (ptr, _guard) = alibi_slopes.device_ptr(&stream);
            ptr as *const core::ffi::c_void
        } else {
            std::ptr::null()
        };

        // if window_size_left > self.max_seqlen_k or None => -1
        let mut window_size_left = self
            .window_size_left
            .filter(|v| v <= &seqlen_k)
            .map(|v| v as i32)
            .unwrap_or(-1);

View on GitHub (pinned to d5fee525bf)

Solutions

1. Move the slopes tensor to the CUDA device with .to_device(&cuda_device) before the call
2. Ensure the whole model (including precomputed buffers like alibi slopes) is loaded on the same CUDA device
3. Check the tensor's device (t.device()) equals q.device() before calling

Example fix

// before
let slopes = Tensor::from_vec(vec![..], (num_heads,), &Device::Cpu)?;
// after
let slopes = Tensor::from_vec(vec![..], (num_heads,), &Device::Cpu)?.to_device(&dev)?;

fn ensure_cuda(t: &Tensor) -> candle::Result<Tensor> {
    if t.device().is_cuda() { Ok(t.clone()) } else { t.to_device(&Device::new_cuda(0)?) }
}
let slopes = ensure_cuda(&slopes)?;

fn is_cuda(t: &Tensor) -> bool { t.device().is_cuda() }

let slopes = slopes.to_device(q.device())?; // unify devices before call
match flash_attn(&q, &k, &v, Some(&slopes), scale, causal) {
    Err(e) if e.to_string().contains("must be a cuda tensor") => { /* move tensors and retry once */ }
    r => r?,
}

When it happens

Trigger: Passing an alibi_slopes tensor that was created on / still resides on the CPU device while q/k/v are CUDA tensors; also occurs if the tensor's dtype is not f32, since as_cuda_slice::<f32>() would fail first, but a non-CUDA storage always triggers this bail.

Common situations: Creating slopes with Tensor::zeros on Device::Cpu by mistake, loading slopes from a checkpoint mapped to CPU and forgetting .to_device(&cuda_device), mixed-device pipelines after moving only q/k/v to GPU.

Related errors

• seqlens_k must be a cuda tensor
• seqlens_q must be a cuda tensor
• seqlens_k must be a cuda tensor
• block_table must be a cuda tensor
• Invalid quantize storage locations do not match

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/be733d054a5e97c1. Report an issue: GitHub.