huggingface/candle · error

mismatch on matmul dim {self_shape:?} {:?}

Error message

mismatch on matmul dim {self_shape:?} {:?}

What it means

After parsing the rhs shape, dmmv checks that the contraction dimension k of the input equals the quantized weight's k. If ncols != k, candle bails reporting both shapes. This is the classic inner-dimension mismatch between the quantized matrix and the input operand.

Source

Thrown at candle-core/src/quantized/cuda.rs:919

    fn dequantize_matmul_vec(
        &self,
        self_shape: &crate::Shape,
        rhs: &CudaStorage,
        rhs_l: &crate::Layout,
    ) -> Result<(CudaStorage, crate::Shape)> {
        let (nrows, ncols) = self_shape.dims2()?;
        let rhs = rhs.as_cuda_slice::<f32>()?;
        let rhs = match rhs_l.contiguous_offsets() {
            Some((o1, o2)) => rhs.slice(o1..o2),
            None => Err(crate::Error::RequiresContiguous { op: "dmmv" }.bt())?,
        };
        let (b_size, k) = match rhs_l.shape().dims() {
            [b, m, k] => (b * m, *k),
            [b, k] => (*b, *k),
            _ => crate::bail!("unexpected rhs shape in dmmv {:?}", rhs_l.shape()),
        };
        if ncols != k {
            crate::bail!("mismatch on matmul dim {self_shape:?} {:?}", rhs_l.shape())
        }

        let out = if FORCE_DMMV.load(std::sync::atomic::Ordering::Relaxed) {
            dequantize_mul_mat_vec(&self.data, &rhs, self.dtype, ncols, nrows, self.device())?
        } else {
            mul_mat_vec_via_q8_1(
                &self.data,
                &rhs,
                self.dtype,
                ncols,
                nrows,
                b_size,
                self.device(),
            )?
        };
        let mut out_shape = rhs_l.shape().dims().to_vec();
        out_shape.pop();
        out_shape.push(nrows);

View on GitHub (pinned to d5fee525bf)

Solutions

1. Fix the input size so its last dim equals the quantized weight's k (reconfigure the model).
2. Load weights and config from the same model version so hidden sizes agree.
3. Add a projection/linear layer to map the input to the expected k.
4. Print both shapes from the message and correct the pipeline around the offending layer.

Example fix

// before: input k2=768 vs weight k=1024
let out = qw.forward(&x)?;
// after: project input to the expected dim first
let proj = linear_in.to_dtype(DType::F32)?; // or use a matching input
let out = qw.forward(&proj.forward(&x)?)?;

let k = qw.shape()?[1]; // contraction dim of quantized weight
assert_eq!(input.dim(candle_core::D::Minus1)?, k,
    "input last dim must equal quantized weight k={k}");

match qw.forward(&input) {
    Err(e) if e.to_string().contains("mismatch on matmul dim") => {
        anyhow::bail!("layer config/weight mismatch, check hidden sizes: {e}")
    }
    r => r?,
}

When it happens

Trigger: Calling quantized matmul/fwd where input last dim k2 differs from the weight's second dim k — e.g. feeding features of the wrong size to a layer, using weights from a different config (hidden size changed between checkpoint and runtime config).

Common situations: Checkpoint/config mismatch (model dimension edited in config.json-like code); connecting layers with different hidden sizes; feeding logits/hidden states of a different model into a quantized layer.

Related errors

• unexpected rhs shape in dmmv {:?}
• unexpected shape for input {s:?}
• unexpected lhs length {} {mkn:?}
• The given quantized dtype {:?} is not supported for indexed_
• only f32 can be quantized

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/cbc1bcf441484e9a. Report an issue: GitHub.