huggingface/candle · error

weight rank ({}) must be <= 4

Error message

weight rank ({}) must be <= 4

What it means

The Metal quantized matmul kernel pads the weight layout into a fixed 4D contiguous layout; tensors with rank greater than 4 cannot be expressed, so this is raised when the quantized weight shape rank exceeds 4.

Source

Thrown at candle-core/src/quantized/metal.rs:393

        let last_k = dst_shape.pop().unwrap();
        if last_k != k {
            crate::bail!("input tensor {layout:?} incompatible with {:?}", self_shape)
        }
        dst_shape.push(n);
        let dst_shape = Shape::from(dst_shape);
        let device = storage.device().clone();
        let dst = device
            .new_buffer_builder()
            .with_size_for(dst_shape.elem_count(), DType::F32)
            .with_label("qmatmul")
            .build()?;
        let encoder = device.command_encoder()?;

        assert_eq!(storage.dtype(), DType::F32);

        if self_shape.rank() > 4 {
            crate::bail!("weight rank ({}) must be <= 4", self_shape.rank())
        }
        let src0_l = crate::Layout::contiguous(
            [vec![1; 4 - self_shape.rank()], self_shape.dims().to_vec()].concat(),
        );
        let src0_stride = src0_l
            .stride()
            .iter()
            .map(|x| {
                (*x as f32 * (self.dtype.type_size() as f32 / self.dtype.block_size() as f32))
                    as usize
            })
            .collect::<Vec<_>>();

        if src_shape.rank() > 4 {
            crate::bail!("weight rank ({}) must be <= 4", src_shape.rank())
        }
        let src1_l = crate::Layout::contiguous(
            [vec![1; 4 - src_shape.rank()], src_shape.dims().to_vec()].concat(),

View on GitHub (pinned to d5fee525bf)

Solutions

1. Reshape the weight to at most 4D before quantization or before the matmul.
2. Merge extra leading dimensions into a single batch dimension (e.g. [a,b,c,m,k] -> [a*b*c,m,k]).
3. Perform the op on CPU/CUDA if you genuinely need higher-rank quantized weights.

Example fix

// before
let w = qtensor.reshape((1, 2, 3, 4, 768 * 4))?; // rank 5
// after
let w = qtensor.reshape((6, 768 * 4))?; // merge leading dims, rank <= 4

fn ensure_weight_rank(w: &candle_core::quantized::QTensor) -> candle_core::Result<()> {
    if w.rank() > 4 { candle_core::bail!("weight rank {} > 4", w.rank()); }
    Ok(())
}

fn weight_rank_ok(rank: usize) -> bool { rank <= 4 }

When it happens

Trigger: QMatMul::fwd on Metal with a quantized weight whose shape has more than 4 dimensions (self_shape.rank() > 4).

Common situations: Constructing 5D+ quantized weights for exotic batching (e.g. video attention with extra head/tile dims); reshaping a quantized tensor and accidentally keeping quant blocks as a dimension.

Understand the failure class

Background: Tensor shape mismatch errors ("must have shape", "expected shape ... got ..."): when tensor dimensions disagree with what an op or layer was told to expect — this error's family across 6 libraries.

Related errors

• input rank ({}) must be >= weight rank ({})
• Invalid quantize storage locations do not match
• not implemented
• input is not a f32 tensor
• unsupported const-set f8e4m3

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/8acbd4d9451eba46. Report an issue: GitHub.