What causes "unexpected rhs shape in dmmv {:?}" in huggingface/candle?

Reshape/unsqueeze the input to [b, k] or [b, m, k]: let x = x.unsqueeze(0)?;

unexpected rhs shape in dmmv {:?}

Error message

unexpected rhs shape in dmmv {:?}

What it means

dequantize_matmul_vec (the dmmv path for quantized CUDA matmul-vec) accepts a rhs of shape [b, k] or [b, m, k]. Any other rank triggers this bail with the offending shape. It is a rank check on the input operand of the quantized matrix-vector product.

Source

Thrown at candle-core/src/quantized/cuda.rs:916

}

impl QCudaStorage {
    fn dequantize_matmul_vec(
        &self,
        self_shape: &crate::Shape,
        rhs: &CudaStorage,
        rhs_l: &crate::Layout,
    ) -> Result<(CudaStorage, crate::Shape)> {
        let (nrows, ncols) = self_shape.dims2()?;
        let rhs = rhs.as_cuda_slice::<f32>()?;
        let rhs = match rhs_l.contiguous_offsets() {
            Some((o1, o2)) => rhs.slice(o1..o2),
            None => Err(crate::Error::RequiresContiguous { op: "dmmv" }.bt())?,
        };
        let (b_size, k) = match rhs_l.shape().dims() {
            [b, m, k] => (b * m, *k),
            [b, k] => (*b, *k),
            _ => crate::bail!("unexpected rhs shape in dmmv {:?}", rhs_l.shape()),
        };
        if ncols != k {
            crate::bail!("mismatch on matmul dim {self_shape:?} {:?}", rhs_l.shape())
        }

        let out = if FORCE_DMMV.load(std::sync::atomic::Ordering::Relaxed) {
            dequantize_mul_mat_vec(&self.data, &rhs, self.dtype, ncols, nrows, self.device())?
        } else {
            mul_mat_vec_via_q8_1(
                &self.data,
                &rhs,
                self.dtype,
                ncols,
                nrows,
                b_size,
                self.device(),
            )?
        };

View on GitHub (pinned to d5fee525bf)

Solutions

Reshape/unsqueeze the input to [b, k] or [b, m, k]: let x = x.unsqueeze(0)?;
Add or remove a batch dimension so the input is 2-D or 3-D.
Check the rank of the input at the call site before the matmul.
Squeeze an extra leading dimension if a rank-4 tensor was passed unintentionally.

Example fix

// before
let out = qw.forward(&vec_1d)?;           // shape [k]
// after
let out = qw.forward(&vec_1d.unsqueeze(0)?)?; // shape [1, k]

Defensive patterns

Strategy: validation

Validate before calling

let dims = input.dims();
if !(dims.len() == 2 || dims.len() == 3) {
    anyhow::bail!("dmmv input must be [b,k] or [b,m,k], got {dims:?}");
}
let input = if dims.len() == 1 { input.unsqueeze(0)? } else { input };

Try / catch

match qw.forward(&input) {
    Err(e) if e.to_string().contains("unexpected rhs shape in dmmv") => {
        let fixed = if input.dims().len() == 1 { input.unsqueeze(0)? } else { input.flatten_to(2)? };
        qw.forward(&fixed)?
    }
    r => r?,
}

Prevention

Keep quantized layer inputs at rank 2 or 3
Unsqueeze batch dim for single vectors
Flatten attention heads before quantized matmuls

When it happens

Trigger: Calling QTensor::fwd/matmul with an input tensor of rank 1 ([k]), rank 3+ beyond [b,m,k], or other shapes — e.g. feeding a 1-D vector without unsqueezing a batch dim.

Common situations: Passing a single token embedding as a 1-D tensor to a quantized layer; reshaping mistakes between the hidden states and quantized weight; model code assuming 3-D inputs while caller supplies 2-D of wrong rank patterns (e.g. [k] or [.., a, b, c]).

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/3737d2782b0ec550. Report an issue: GitHub.