What causes "different inner dimensions in broadcast matmul {lhs:?} {rhs:?}" in huggingface/candle?

Fix the contraction dimension: transpose the RHS (or LHS) so inner dims match, e.g. x.matmul(&w.t()?)

different inner dimensions in broadcast matmul {lhs:?} {rhs:

Error message

different inner dimensions in broadcast matmul {lhs:?} {rhs:?}

What it means

After extracting the trailing 2x2 matrix dims, broadcast_shape_matmul checks that the inner (contraction) dimensions match: lhs last dim must equal rhs second-to-last dim. If lhs_k != rhs_k the matmul is mathematically undefined and the library bails, echoing both shapes.

Source

Thrown at candle-core/src/shape.rs:239

                    op,
                }
                .bt())?
            }
        }
        Ok(Shape::from(bcast_dims))
    }

    pub(crate) fn broadcast_shape_matmul(&self, rhs: &Self) -> Result<(Shape, Shape)> {
        let lhs = self;
        let lhs_dims = lhs.dims();
        let rhs_dims = rhs.dims();
        if lhs_dims.len() < 2 || rhs_dims.len() < 2 {
            crate::bail!("only 2d matrixes are supported {lhs:?} {rhs:?}")
        }
        let (m, lhs_k) = (lhs_dims[lhs_dims.len() - 2], lhs_dims[lhs_dims.len() - 1]);
        let (rhs_k, n) = (rhs_dims[rhs_dims.len() - 2], rhs_dims[rhs_dims.len() - 1]);
        if lhs_k != rhs_k {
            crate::bail!("different inner dimensions in broadcast matmul {lhs:?} {rhs:?}")
        }

        let lhs_b = Self::from(&lhs_dims[..lhs_dims.len() - 2]);
        let rhs_b = Self::from(&rhs_dims[..rhs_dims.len() - 2]);
        let bcast = lhs_b.broadcast_shape_binary_op(&rhs_b, "broadcast_matmul")?;
        let bcast_dims = bcast.dims();

        let bcast_lhs = [bcast_dims, &[m, lhs_k]].concat();
        let bcast_rhs = [bcast_dims, &[rhs_k, n]].concat();
        Ok((Shape::from(bcast_lhs), Shape::from(bcast_rhs)))
    }
}

pub trait Dim {
    fn to_index(&self, shape: &Shape, op: &'static str) -> Result<usize>;
    fn to_index_plus_one(&self, shape: &Shape, op: &'static str) -> Result<usize>;
}

View on GitHub (pinned to d5fee525bf)

Solutions

Fix the contraction dimension: transpose the RHS (or LHS) so inner dims match, e.g. x.matmul(&w.t()?)
Verify weight shapes match the model config's hidden size; reload the correct checkpoint
Print both .dims() and align reshapes so lhs.dims()[-1] == rhs.dims()[-2]

Example fix

// before
let y = x.matmul(&w)?;        // x: (b, 768), w: (1024, 768)
// after
let y = x.matmul(&w.t()?)?;   // w.t(): (768, 1024)

Defensive patterns

Strategy: validation

Validate before calling

let (l, r) = (lhs.dims(), rhs.dims());
if l[l.len()-1] != r[r.len()-2] {
    return Err(anyhow::anyhow!("matmul inner dims differ: {:?} x {:?}", l, r));
}

Try / catch

let y = lhs.matmul(&rhs).map_err(|e| {
    if e.to_string().contains("inner dimensions") {
        anyhow::anyhow!("check weight transpose/hidden size: {:?} vs {:?}", lhs.dims(), rhs.dims())
    } else { e.into() }
})?;

Prevention

Verify lhs.dims()[-1] == rhs.dims()[-2] before every matmul
Check checkpoint weight shapes against model config hidden sizes
Remember candle matmul does not transpose: add .t() explicitly where needed

When it happens

Trigger: Calling matmul/broadcast_matmul with shapes like (m, k1) x (k2, n) where k1 != k2, or batched variants where the last two dims are incompatible.

Common situations: Mixing layers with wrong hidden sizes (e.g. 768 vs 1024 weights); transposing one operand incorrectly (forgot .t() or applied it twice); loading mismatched checkpoints; reshape mistakes that alter the K dimension.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/ac9ab66be6295273. Report an issue: GitHub.