What causes "unexpected shape for input {s:?}" in huggingface/candle?

Flatten to 3-D first: let x = x.flatten_from(1)?; or reshape to [b, m, k].

unexpected shape for input {s:?}

Error message

unexpected shape for input {s:?}

What it means

dequantize_matmul (the full matmul path) accepts an input of shape [b, m, k2] or [m, k2] (treated as batch 1). Any other shape — rank 1, rank 4+, or non-standard patterns — triggers this bail printing the actual shape. It mirrors the dmmv rank check for the non-vector path.

Source

Thrown at candle-core/src/quantized/cuda.rs:952

        };
        let mut out_shape = rhs_l.shape().dims().to_vec();
        out_shape.pop();
        out_shape.push(nrows);
        Ok((out, out_shape.into()))
    }

    fn dequantize_matmul(
        &self,
        self_shape: &crate::Shape,
        storage: &CudaStorage,
        layout: &crate::Layout,
    ) -> Result<(CudaStorage, crate::Shape)> {
        use crate::backend::BackendStorage;
        let (n, k) = self_shape.dims2()?;
        let (b, m, k2) = match layout.shape().dims() {
            &[b, m, k2] => (b, m, k2),
            &[m, k2] => (1, m, k2),
            s => crate::bail!("unexpected shape for input {s:?}"),
        };
        if k2 != k {
            crate::bail!("mismatch on matmul dim {self_shape:?} {:?}", layout.shape())
        }

        let out = if FORCE_DMMV.load(std::sync::atomic::Ordering::Relaxed) {
            let data_f32 = self.dequantize(n * k)?;
            let rhs_l = crate::Layout::new((k, n).into(), vec![1, k], 0).broadcast_as((b, k, n))?;
            storage.matmul(&data_f32, (b, m, n, k), layout, &rhs_l)?
        } else {
            let storage = storage.as_cuda_slice::<f32>()?;
            let storage = match layout.contiguous_offsets() {
                Some((o1, o2)) => storage.slice(o1..o2),
                None => Err(crate::Error::RequiresContiguous {
                    op: "quantized-matmul",
                }
                .bt())?,
            };

View on GitHub (pinned to d5fee525bf)

Solutions

Flatten to 3-D first: let x = x.flatten_from(1)?; or reshape to [b, m, k].
Unsqueeze a batch dim for rank-1 input.
Fold head/extra dims into the batch dimension before the matmul.
Verify input rank with x.dims().len() before calling the quantized op.

Example fix

// before
let out = qw.forward(&attn)?; // [b, heads, m, k]
// after
let (b, h, m, k) = attn.dims4()?;
let out = qw.forward(&attn.reshape((b * h, m, k))?)?;

Defensive patterns

Strategy: validation

Validate before calling

let dims = input.dims();
if !(dims.len() == 2 || dims.len() == 3) {
    let input = if dims.len() == 1 { input.unsqueeze(0)? }
        else { input.flatten_to(2)? }; // fold extra dims into batch
}
qw.forward(&input)?;

Try / catch

match qw.forward(&input) {
    Err(e) if e.to_string().contains("unexpected shape for input") => {
        let x = if input.dims().len() == 1 { input.unsqueeze(0)? } else { input.flatten_to(2)? };
        qw.forward(&x)?
    }
    r => r?,
}

Prevention

Reshape rank-4 attention tensors to [b*heads, m, k] before quantized matmuls
Unsqueeze 1-D inputs to add a batch dim
Check rank with dims().len() in a helper before any quantized op

When it happens

Trigger: Calling QTensor::fwd with an input of rank != 2/3 — e.g. a [k] vector, a [b, heads, m, k] attention tensor, or a [b, s, p, k] tensor from an earlier reshape.

Common situations: Passing multi-head attention tensors (rank 4) directly to a quantized layer without flattening heads; feeding a raw 1-D embedding; pipeline reshaping bugs where an extra dim remains.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/97c5af48a035d958. Report an issue: GitHub.