What causes "input rank ({}) must be >= weight rank ({})" in huggingface/candle?

Reshape the input so its rank is >= the weight rank (e.g. unsqueeze batch dims) before the quantized matmul.

input rank ({}) must be >= weight rank ({})

Error message

input rank ({}) must be >= weight rank ({})

What it means

Thrown by the Metal quantized matmul path when the input (src) tensor's rank is lower than the quantized weight tensor's rank. The kernel needs the input to be at least as high-dimensional as the weight so the last two dims line up for matrix multiplication.

Source

Thrown at candle-core/src/quantized/metal.rs:365

        storage: &MetalStorage,
        layout: &crate::Layout,
    ) -> Result<(MetalStorage, Shape)> {
        use crate::MetalError;

        if !layout.is_contiguous() {
            crate::bail!("input tensor is not contiguous {layout:?}")
        }
        let src_shape = layout.shape();
        // self is transposed so n is first then k.
        if src_shape.rank() < 2 {
            crate::bail!("input tensor has only one dimension {layout:?}")
        }
        let n = self_shape.dim(D::Minus2)?;
        let k = self_shape.dim(D::Minus1)?;
        let mut dst_shape = src_shape.dims().to_vec();

        if src_shape.rank() < self_shape.rank() {
            crate::bail!(
                "input rank ({}) must be >= weight rank ({})",
                src_shape.rank(),
                self_shape.rank()
            )
        }

        if src_shape.dim(D::Minus2)? == 1 {
            return self.fwd_mv(self_shape, storage, layout);
        }

        let last_k = dst_shape.pop().unwrap();
        if last_k != k {
            crate::bail!("input tensor {layout:?} incompatible with {:?}", self_shape)
        }
        dst_shape.push(n);
        let dst_shape = Shape::from(dst_shape);
        let device = storage.device().clone();
        let dst = device

View on GitHub (pinned to d5fee525bf)

Solutions

Reshape the input so its rank is >= the weight rank (e.g. unsqueeze batch dims) before the quantized matmul.
Flatten/reshape the weight to 2D [out, in] so a rank-2 input is sufficient.
Check the shapes right before the QMatMul call and align them.
If you don't need Metal, run the same op on CPU where the rank constraint is different.

Example fix

// before
let y = qmatmul.forward(&x)?; // x rank 2, weight rank 3
// after
let x = x.unsqueeze(0)?; // align input rank with weight rank
let y = qmatmul.forward(&x)?;

Defensive patterns

Strategy: validation

Validate before calling

fn ensure_input_rank_ok(input_rank: usize, weight: &candle_core::quantized::QTensor) -> candle_core::Result<()> {
    if input_rank < weight.rank() {
        candle_core::bail!("input rank {} < weight rank {}", input_rank, weight.rank());
    }
    Ok(())
}

Type guard

fn input_rank_ok(input_rank: usize, weight_rank: usize) -> bool { input_rank >= weight_rank }

Prevention

Match input rank to weight rank with unsqueeze before quantized matmuls
Keep quantized weights 2D [out, in] unless batching requires more
Log shapes of weights and activations at model-load time

When it happens

Trigger: Calling QMatMul::fwd (or quantized matmul) on Metal with a weight of rank >= 3 (e.g. reshaped to [b, m, k]) while the input is rank 1 or 2.

Common situations: Batching a quantized layer where the weight was expanded/reshaped but the activation was not; passing a 1D vector into a rank-3 quantized weight matmul on the Metal backend.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/610a27c6c3a83025. Report an issue: GitHub.