{"record":{"id":"7962ddc3b9a51a08","repo":"huggingface/candle","slug":"expected-f32-f16","errorCode":null,"errorMessage":"Expected f32/f16","messagePattern":"Expected f32/f16","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-core/src/quantized/mod.rs","lineNumber":991,"sourceCode":"                    (dst_shape.elem_count() / n, k, n),\n                    slice,\n                    &mut dst_storage,\n                )?;\n                Ok((crate::CpuStorage::F32(dst_storage), dst_shape))\n            }\n            DType::F16 => {\n                let slice = storage.as_slice::<f16>()?;\n                let slice =\n                    &slice[layout.start_offset()..layout.start_offset() + src_shape.elem_count()];\n                let mut dst_storage = vec![f16::ZERO; dst_shape.elem_count()];\n                self_storage.matmul_t_f16(\n                    (dst_shape.elem_count() / n, k, n),\n                    slice,\n                    &mut dst_storage,\n                )?;\n                Ok((crate::CpuStorage::F16(dst_storage), dst_shape))\n            }\n            _ => crate::bail!(\"Expected f32/f16\"),\n        }\n    }\n\n    fn metal_fwd(\n        &self,\n        storage: &crate::MetalStorage,\n        layout: &crate::Layout,\n    ) -> Result<(crate::MetalStorage, Shape)> {\n        let self_storage = match &self.storage {\n            QStorage::Metal(metal) => metal,\n            _ => unreachable!(\"Cannot call metal matmul on non metal QTensor\"),\n        };\n        self_storage.fwd(&self.shape, storage, layout)\n    }\n\n    fn cuda_fwd(\n        &self,\n        storage: &crate::CudaStorage,","sourceCodeStart":973,"sourceCodeEnd":1009,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-core/src/quantized/mod.rs#L973-L1009","documentation":"This error is thrown by the quantized matmul fast-path ( QuantizedMatMul::cpu_fwd ) when the LHS activation tensor storage is neither F32 nor F16. The quantized CPU kernels (gemmv/gemm for QKK types) are only implemented for f32 and f16 activations, so any other dtype (e.g. bf16, u8, i64) reaching this op on CPU bails.","triggerScenarios":"Calling QMatMul::forward / Tensor::quantized_matmul on CPU with a left-hand tensor whose dtype is not F32 or F16, e.g. after .to_dtype(DType::BF16) or passing an integer/quantized tensor as the activation input.","commonSituations":"Loading a GGUF model and casting activations to bf16 for memory savings; accidentally feeding a quantized or integer tensor as the hidden-state input; mixing a dtype conversion (bf16) with quantized inference in candle.","solutions":["Convert the activation tensor to DType::F32 before the quantized matmul: tensor.to_dtype(DType::F32)?","Alternatively convert to DType::F16 if the surrounding model uses half precision","Check where the tensor dtype changed (e.g. a .to_dtype call or model config dtype) and keep activations f32/f16 for quantized layers"],"exampleFix":"// before\nlet x = x.to_dtype(DType::BF16)?;\nlet y = qmatmul.forward(&x)?;\n// after\nlet x = x.to_dtype(DType::F32)?;\nlet y = qmatmul.forward(&x)?;","handlingStrategy":"validation","validationCode":"if !matches!(x.dtype(), candle::DType::F32 | candle::DType::F16) {\n    x = x.to_dtype(candle::DType::F32)?;\n}\nlet y = qmatmul.forward(&x)?;","typeGuard":"fn is_quantized_mm_ok(t: &candle_core::Tensor) -> bool {\n    matches!(t.dtype(), candle_core::DType::F32 | candle_core::DType::F16)\n}","tryCatchPattern":"let y = qmatmul.forward(&x).or_else(|_| qmatmul.forward(&x.to_dtype(candle::DType::F32)?))?;","preventionTips":["Keep activation tensors F32 (or F16) end-to-end in quantized GGUF inference","Cast to bf16 only after quantized matmul layers, if needed at all","Assert dtype with a debug_assert before each QMatMul call during development"],"tags":["candle","quantized","matmul","dtype"],"backgroundTag":"unsupported-dtype-for-op","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}