{"record":{"id":"3737d2782b0ec550","repo":"huggingface/candle","slug":"unexpected-rhs-shape-in-dmmv","errorCode":null,"errorMessage":"unexpected rhs shape in dmmv {:?}","messagePattern":"unexpected rhs shape in dmmv (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-core/src/quantized/cuda.rs","lineNumber":916,"sourceCode":"}\n\nimpl QCudaStorage {\n    fn dequantize_matmul_vec(\n        &self,\n        self_shape: &crate::Shape,\n        rhs: &CudaStorage,\n        rhs_l: &crate::Layout,\n    ) -> Result<(CudaStorage, crate::Shape)> {\n        let (nrows, ncols) = self_shape.dims2()?;\n        let rhs = rhs.as_cuda_slice::<f32>()?;\n        let rhs = match rhs_l.contiguous_offsets() {\n            Some((o1, o2)) => rhs.slice(o1..o2),\n            None => Err(crate::Error::RequiresContiguous { op: \"dmmv\" }.bt())?,\n        };\n        let (b_size, k) = match rhs_l.shape().dims() {\n            [b, m, k] => (b * m, *k),\n            [b, k] => (*b, *k),\n            _ => crate::bail!(\"unexpected rhs shape in dmmv {:?}\", rhs_l.shape()),\n        };\n        if ncols != k {\n            crate::bail!(\"mismatch on matmul dim {self_shape:?} {:?}\", rhs_l.shape())\n        }\n\n        let out = if FORCE_DMMV.load(std::sync::atomic::Ordering::Relaxed) {\n            dequantize_mul_mat_vec(&self.data, &rhs, self.dtype, ncols, nrows, self.device())?\n        } else {\n            mul_mat_vec_via_q8_1(\n                &self.data,\n                &rhs,\n                self.dtype,\n                ncols,\n                nrows,\n                b_size,\n                self.device(),\n            )?\n        };","sourceCodeStart":898,"sourceCodeEnd":934,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-core/src/quantized/cuda.rs#L898-L934","documentation":"dequantize_matmul_vec (the dmmv path for quantized CUDA matmul-vec) accepts a rhs of shape [b, k] or [b, m, k]. Any other rank triggers this bail with the offending shape. It is a rank check on the input operand of the quantized matrix-vector product.","triggerScenarios":"Calling QTensor::fwd/matmul with an input tensor of rank 1 ([k]), rank 3+ beyond [b,m,k], or other shapes — e.g. feeding a 1-D vector without unsqueezing a batch dim.","commonSituations":"Passing a single token embedding as a 1-D tensor to a quantized layer; reshaping mistakes between the hidden states and quantized weight; model code assuming 3-D inputs while caller supplies 2-D of wrong rank patterns (e.g. [k] or [.., a, b, c]).","solutions":["Reshape/unsqueeze the input to [b, k] or [b, m, k]: let x = x.unsqueeze(0)?;","Add or remove a batch dimension so the input is 2-D or 3-D.","Check the rank of the input at the call site before the matmul.","Squeeze an extra leading dimension if a rank-4 tensor was passed unintentionally."],"exampleFix":"// before\nlet out = qw.forward(&vec_1d)?;           // shape [k]\n// after\nlet out = qw.forward(&vec_1d.unsqueeze(0)?)?; // shape [1, k]","handlingStrategy":"validation","validationCode":"let dims = input.dims();\nif !(dims.len() == 2 || dims.len() == 3) {\n    anyhow::bail!(\"dmmv input must be [b,k] or [b,m,k], got {dims:?}\");\n}\nlet input = if dims.len() == 1 { input.unsqueeze(0)? } else { input };","typeGuard":null,"tryCatchPattern":"match qw.forward(&input) {\n    Err(e) if e.to_string().contains(\"unexpected rhs shape in dmmv\") => {\n        let fixed = if input.dims().len() == 1 { input.unsqueeze(0)? } else { input.flatten_to(2)? };\n        qw.forward(&fixed)?\n    }\n    r => r?,\n}","preventionTips":["Keep quantized layer inputs at rank 2 or 3","Unsqueeze batch dim for single vectors","Flatten attention heads before quantized matmuls"],"tags":["cuda","quantization","matmul","shape","rank"],"backgroundTag":"shape-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}