{"record":{"id":"cbc1bcf441484e9a","repo":"huggingface/candle","slug":"mismatch-on-matmul-dim-self-shape","errorCode":null,"errorMessage":"mismatch on matmul dim {self_shape:?} {:?}","messagePattern":"mismatch on matmul dim (.+?) (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-core/src/quantized/cuda.rs","lineNumber":919,"sourceCode":"    fn dequantize_matmul_vec(\n        &self,\n        self_shape: &crate::Shape,\n        rhs: &CudaStorage,\n        rhs_l: &crate::Layout,\n    ) -> Result<(CudaStorage, crate::Shape)> {\n        let (nrows, ncols) = self_shape.dims2()?;\n        let rhs = rhs.as_cuda_slice::<f32>()?;\n        let rhs = match rhs_l.contiguous_offsets() {\n            Some((o1, o2)) => rhs.slice(o1..o2),\n            None => Err(crate::Error::RequiresContiguous { op: \"dmmv\" }.bt())?,\n        };\n        let (b_size, k) = match rhs_l.shape().dims() {\n            [b, m, k] => (b * m, *k),\n            [b, k] => (*b, *k),\n            _ => crate::bail!(\"unexpected rhs shape in dmmv {:?}\", rhs_l.shape()),\n        };\n        if ncols != k {\n            crate::bail!(\"mismatch on matmul dim {self_shape:?} {:?}\", rhs_l.shape())\n        }\n\n        let out = if FORCE_DMMV.load(std::sync::atomic::Ordering::Relaxed) {\n            dequantize_mul_mat_vec(&self.data, &rhs, self.dtype, ncols, nrows, self.device())?\n        } else {\n            mul_mat_vec_via_q8_1(\n                &self.data,\n                &rhs,\n                self.dtype,\n                ncols,\n                nrows,\n                b_size,\n                self.device(),\n            )?\n        };\n        let mut out_shape = rhs_l.shape().dims().to_vec();\n        out_shape.pop();\n        out_shape.push(nrows);","sourceCodeStart":901,"sourceCodeEnd":937,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-core/src/quantized/cuda.rs#L901-L937","documentation":"After parsing the rhs shape, dmmv checks that the contraction dimension k of the input equals the quantized weight's k. If ncols != k, candle bails reporting both shapes. This is the classic inner-dimension mismatch between the quantized matrix and the input operand.","triggerScenarios":"Calling quantized matmul/fwd where input last dim k2 differs from the weight's second dim k — e.g. feeding features of the wrong size to a layer, using weights from a different config (hidden size changed between checkpoint and runtime config).","commonSituations":"Checkpoint/config mismatch (model dimension edited in config.json-like code); connecting layers with different hidden sizes; feeding logits/hidden states of a different model into a quantized layer.","solutions":["Fix the input size so its last dim equals the quantized weight's k (reconfigure the model).","Load weights and config from the same model version so hidden sizes agree.","Add a projection/linear layer to map the input to the expected k.","Print both shapes from the message and correct the pipeline around the offending layer."],"exampleFix":"// before: input k2=768 vs weight k=1024\nlet out = qw.forward(&x)?;\n// after: project input to the expected dim first\nlet proj = linear_in.to_dtype(DType::F32)?; // or use a matching input\nlet out = qw.forward(&proj.forward(&x)?)?;","handlingStrategy":"validation","validationCode":"let k = qw.shape()?[1]; // contraction dim of quantized weight\nassert_eq!(input.dim(candle_core::D::Minus1)?, k,\n    \"input last dim must equal quantized weight k={k}\");","typeGuard":null,"tryCatchPattern":"match qw.forward(&input) {\n    Err(e) if e.to_string().contains(\"mismatch on matmul dim\") => {\n        anyhow::bail!(\"layer config/weight mismatch, check hidden sizes: {e}\")\n    }\n    r => r?,\n}","preventionTips":["Load config and weights from the same model revision","Assert last-dim equality at layer boundaries in custom pipelines","Re-export GGUF files after any config dimension change"],"tags":["cuda","quantization","matmul","shape-mismatch","dimension"],"backgroundTag":"matmul-dimension-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}