huggingface/candle · error

weight rank ( ) must be <= 4

Error message

weight rank ({}) must be <= 4

What it means

The Metal quantized matmul kernel pads the weight layout into a fixed 4D contiguous layout; tensors with rank greater than 4 cannot be expressed, so this is raised when the quantized weight shape rank exceeds 4.

Solutions

1. Reshape the weight to at most 4D before quantization or before the matmul.
2. Merge extra leading dimensions into a single batch dimension (e.g. [a,b,c,m,k] -> [a*b*c,m,k]).
3. Perform the op on CPU/CUDA if you genuinely need higher-rank quantized weights.

Example fix

// before
let w = qtensor.reshape((1, 2, 3, 4, 768 * 4))?; // rank 5
// after
let w = qtensor.reshape((6, 768 * 4))?; // merge leading dims, rank <= 4

fn ensure_weight_rank(w: &candle_core::quantized::QTensor) -> candle_core::Result<()> {
    if w.rank() > 4 { candle_core::bail!("weight rank {} > 4", w.rank()); }
    Ok(())
}

fn weight_rank_ok(rank: usize) -> bool { rank <= 4 }

When it happens

Trigger: QMatMul::fwd on Metal with a quantized weight whose shape has more than 4 dimensions (self_shape.rank() > 4).

Common situations: Constructing 5D+ quantized weights for exotic batching (e.g. video attention with extra head/tile dims); reshaping a quantized tensor and accidentally keeping quant blocks as a dimension.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

• input rank ( ) must be >= weight rank ( )
• Invalid quantize storage locations do not match
• not implemented
• all q, k, v dtypes must match.
• convtr1d: shape mismatch on c_in

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/8acbd4d9451eba46. Report an issue: GitHub.

Appendix: source

Thrown at candle-core/src/quantized/metal.rs:393

        let last_k = dst_shape.pop().unwrap();
        if last_k != k {
            crate::bail!("input tensor {layout:?} incompatible with {:?}", self_shape)
        }
        dst_shape.push(n);
        let dst_shape = Shape::from(dst_shape);
        let device = storage.device().clone();
        let dst = device
            .new_buffer_builder()
            .with_size_for(dst_shape.elem_count(), DType::F32)
            .with_label("qmatmul")
            .build()?;
        let encoder = device.command_encoder()?;

        assert_eq!(storage.dtype(), DType::F32);

        if self_shape.rank() > 4 {
            crate::bail!("weight rank ({}) must be <= 4", self_shape.rank())
        }
        let src0_l = crate::Layout::contiguous(
            [vec![1; 4 - self_shape.rank()], self_shape.dims().to_vec()].concat(),
        );
        let src0_stride = src0_l
            .stride()
            .iter()
            .map(|x| {
                (*x as f32 * (self.dtype.type_size() as f32 / self.dtype.block_size() as f32))
                    as usize
            })
            .collect::<Vec<_>>();

        if src_shape.rank() > 4 {
            crate::bail!("weight rank ({}) must be <= 4", src_shape.rank())
        }
        let src1_l = crate::Layout::contiguous(
            [vec![1; 4 - src_shape.rank()], src_shape.dims().to_vec()].concat(),

View on GitHub (pinned to d5fee525bf)