What causes "shape mismatch alibi_slopes {:?}, expected {:?}" in huggingface/candle?

Reshape alibi_slopes to a 1D tensor of exactly num_heads elements before calling

shape mismatch alibi_slopes {:?}, expected {:?}

Error message

shape mismatch alibi_slopes {:?}, expected {:?}

What it means

candle-flash-attn throws this when the optional alibi_slopes tensor's shape does not have exactly `num_heads` elements. ALIBI slopes must provide one slope value per attention head, so the kernel validates dim1 size == num_heads before launch.

Source

Thrown at candle-flash-attn/src/lib.rs:103

        }
        if num_heads % num_heads_k != 0 {
            candle::bail!("number of k/v heads {num_heads_k} must divide number of heads in query {num_heads}")
        }

        let stream = dev.cuda_stream();
        let alibi_slopes_ptr = if let Some(alibi_slopes) = &self.alibi_slopes {
            if alibi_slopes.dtype() != DType::F32 {
                candle::bail!(
                    "DType mismatch alibi_slopes {:?}, expected {:?}",
                    alibi_slopes.dtype(),
                    DType::F32
                );
            }

            let (alibi_slopes, alibi_slopes_layout) = alibi_slopes.storage_and_layout();

            if num_heads != alibi_slopes_layout.shape().dims1()? {
                candle::bail!(
                    "shape mismatch alibi_slopes {:?}, expected {:?}",
                    alibi_slopes_layout.shape(),
                    (num_heads)
                );
            }

            let alibi_slopes = match &*alibi_slopes {
                candle::Storage::Cuda(c) => c.as_cuda_slice::<f32>()?,
                _ => candle::bail!("alibi_slopes must be a cuda tensor"),
            };

            let alibi_slopes = alibi_slopes.slice(alibi_slopes_layout.start_offset()..);

            // Dropping the guard here doesn't seem very safe.
            let (ptr, _guard) = alibi_slopes.device_ptr(&stream);
            ptr as *const core::ffi::c_void
        } else {
            std::ptr::null()

View on GitHub (pinned to d5fee525bf)

Solutions

Reshape alibi_slopes to a 1D tensor of exactly num_heads elements before calling
Regenerate slopes for the current num_heads config (slopes depend on head count)
Remove the alibi_slopes argument if ALIBI is not intended

Example fix

// before
let slopes = Tensor::from_vec(slopes_vec, (1, num_heads), &dev)?;
flash_attn(&q, &k, &v, Some(&slopes), softmax_scale, causal)?
// after
let slopes = Tensor::from_vec(slopes_vec, (num_heads,), &dev)?;
flash_attn(&q, &k, &v, Some(&slopes), softmax_scale, causal)?

Defensive patterns

Strategy: validation

Validate before calling

fn check_alibi(slopes: &Tensor, num_heads: usize) -> candle::Result<()> {
    let dims = slopes.dims1()?; // errors if not 1D
    if dims != num_heads {
        candle::bail!("alibi_slopes len {dims} != num_heads {num_heads}");
    }
    Ok(())
}

Type guard

fn is_valid_alibi(slopes: &Tensor, num_heads: usize) -> bool {
    slopes.dims() == [num_heads]
}

Try / catch

match flash_attn(&q, &k, &v, Some(&slopes), scale, causal) {
    Err(candle::Error::Msg(m)) if m.contains("alibi_slopes") => { /* rebuild slopes with correct num_heads */ }
    r => r?,
}

Prevention

Derive alibi slopes programmatically from num_heads instead of hardcoding
Assert slopes.dim(0) == q.dim(2) (num_heads) in model init
Keep slopes 1D; never add batch/head axes

When it happens

Trigger: Calling flash_attn (or FlashAttnV2 forward) with an `alibi_slopes` tensor whose rank is not 1 or whose length differs from the q tensor's num_heads dimension; `dims1()?` also fails for non-1D tensors and that error propagates similarly.

Common situations: Reusing slopes prepared for a different head count after changing model config (e.g. num_attention_heads), passing a [1, num_heads] or [num_heads, 1] tensor instead of 1D, or passing a wrong slice of a larger slopes tensor.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/7103d96fa2862e2c. Report an issue: GitHub.