{"record":{"id":"8b7f689c8b2bef2b","repo":"huggingface/candle","slug":"only-f32-can-be-quantized","errorCode":null,"errorMessage":"only f32 can be quantized","messagePattern":"only f32 can be quantized","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-core/src/quantized/cuda.rs","lineNumber":708,"sourceCode":"            GgmlDType::Q4K => deq::<crate::quantized::BlockQ4K>(&buffer, block_len, &mut out),\n            GgmlDType::Q5K => deq::<crate::quantized::BlockQ5K>(&buffer, block_len, &mut out),\n            GgmlDType::Q6K => deq::<crate::quantized::BlockQ6K>(&buffer, block_len, &mut out),\n            GgmlDType::Q8K => deq::<crate::quantized::BlockQ8K>(&buffer, block_len, &mut out),\n        }\n\n        self.device\n            .storage_from_cpu_storage(&crate::CpuStorage::F32(out))\n    }\n\n    pub fn dequantize_f16(&self, elem_count: usize) -> Result<CudaStorage> {\n        dequantize_f16(&self.data, self.dtype, elem_count, self.device())\n    }\n\n    pub fn quantize(&mut self, src: &CudaStorage) -> Result<()> {\n        // Run the quantization on cpu.\n        let src = match &src.slice {\n            crate::cuda_backend::CudaStorageSlice::F32(data) => self.device.clone_dtoh(data)?,\n            _ => crate::bail!(\"only f32 can be quantized\"),\n        };\n        let src_len = src.len();\n        let src = crate::Storage::Cpu(crate::CpuStorage::F32(src));\n        let mut qcpu_storage = crate::Device::Cpu.qzeros(src_len, self.dtype)?;\n        qcpu_storage.quantize(&src)?;\n        let data = qcpu_storage.data()?;\n        let padded_len =\n            data.len() + MATRIX_ROW_PADDING * self.dtype.type_size() / self.dtype.block_size();\n        let mut inner = unsafe { self.device.alloc::<u8>(padded_len)? };\n        self.device\n            .memcpy_htod(&*data, &mut inner.slice_mut(..data.len()))?;\n        self.data = PaddedCudaSlice {\n            inner,\n            len: data.len(),\n        };\n        Ok(())\n    }\n","sourceCodeStart":690,"sourceCodeEnd":726,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-core/src/quantized/cuda.rs#L690-L726","documentation":"QQuantized::quantize on CUDA can only convert an f32 CUDA source storage; it downloads the data and runs quantization on the CPU. If the source CudaStorage holds any other element type (e.g. f16, bf16, u8), the match falls through to bail. Quantizing half-precision tensors directly is not supported.","triggerScenarios":"Calling quantize on a CudaQuantizedStorage whose source storage slice is not CudaStorageSlice::F32 — e.g. passing a half-precision (f16/bf16) CUDA tensor to quantize, or a tensor produced by an op that yielded another dtype.","commonSituations":"Quantizing a model whose weights were cast to f16 for CUDA inference; pipeline code that assumes all weights are f32 when they were converted to half; copying quantization code from a CPU example to a CUDA tensor stored in f16.","solutions":["Cast the source tensor to f32 first: let w = tensor.to_dtype(candle_core::DType::F32)?; then call quantize.","Ensure the checkpoint loading path produces f32 weights (avoid automatic f16 casting on device).","Quantize on CPU from an f32 CPU tensor instead.","Check tensor.dtype() == DType::F32 before calling quantize."],"exampleFix":"// before\nqstorage.quantize(&f16_cuda_storage)?;\n// after\nlet f32_storage = tensor.to_dtype(candle_core::DType::F32)?.to_device(&Device::Cuda)?;\nqstorage.quantize(&f32_storage)?;","handlingStrategy":"validation","validationCode":"if tensor.dtype() != candle_core::DType::F32 {\n    let tensor = tensor.to_dtype(candle_core::DType::F32)?;\n}\nqstorage.quantize(&tensor_to_cuda_f32(&tensor)?)?;","typeGuard":"fn is_f32_cuda(t: &candle_core::Tensor) -> bool {\n    t.dtype() == candle_core::DType::F32 && t.device().is_cuda()\n}","tryCatchPattern":"match qstorage.quantize(&src) {\n    Err(e) if e.to_string().contains(\"only f32 can be quantized\") => {\n        let f32_src = src_tensor.to_dtype(candle_core::DType::F32)?;\n        qstorage.quantize(&f32_src)?\n    }\n    r => r?,\n}","preventionTips":["Always cast to f32 before quantizing","Avoid casting whole models to f16 before the quantization step","Assert dtype f32 in the quantization pipeline entry point"],"tags":["cuda","quantization","dtype","f32"],"backgroundTag":"dtype-not-supported","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}