huggingface/candle · error

only f32 can be quantized

Error message

only f32 can be quantized

What it means

QQuantized::quantize on CUDA can only convert an f32 CUDA source storage; it downloads the data and runs quantization on the CPU. If the source CudaStorage holds any other element type (e.g. f16, bf16, u8), the match falls through to bail. Quantizing half-precision tensors directly is not supported.

Source

Thrown at candle-core/src/quantized/cuda.rs:708

            GgmlDType::Q4K => deq::<crate::quantized::BlockQ4K>(&buffer, block_len, &mut out),
            GgmlDType::Q5K => deq::<crate::quantized::BlockQ5K>(&buffer, block_len, &mut out),
            GgmlDType::Q6K => deq::<crate::quantized::BlockQ6K>(&buffer, block_len, &mut out),
            GgmlDType::Q8K => deq::<crate::quantized::BlockQ8K>(&buffer, block_len, &mut out),
        }

        self.device
            .storage_from_cpu_storage(&crate::CpuStorage::F32(out))
    }

    pub fn dequantize_f16(&self, elem_count: usize) -> Result<CudaStorage> {
        dequantize_f16(&self.data, self.dtype, elem_count, self.device())
    }

    pub fn quantize(&mut self, src: &CudaStorage) -> Result<()> {
        // Run the quantization on cpu.
        let src = match &src.slice {
            crate::cuda_backend::CudaStorageSlice::F32(data) => self.device.clone_dtoh(data)?,
            _ => crate::bail!("only f32 can be quantized"),
        };
        let src_len = src.len();
        let src = crate::Storage::Cpu(crate::CpuStorage::F32(src));
        let mut qcpu_storage = crate::Device::Cpu.qzeros(src_len, self.dtype)?;
        qcpu_storage.quantize(&src)?;
        let data = qcpu_storage.data()?;
        let padded_len =
            data.len() + MATRIX_ROW_PADDING * self.dtype.type_size() / self.dtype.block_size();
        let mut inner = unsafe { self.device.alloc::<u8>(padded_len)? };
        self.device
            .memcpy_htod(&*data, &mut inner.slice_mut(..data.len()))?;
        self.data = PaddedCudaSlice {
            inner,
            len: data.len(),
        };
        Ok(())
    }

View on GitHub (pinned to d5fee525bf)

Solutions

1. Cast the source tensor to f32 first: let w = tensor.to_dtype(candle_core::DType::F32)?; then call quantize.
2. Ensure the checkpoint loading path produces f32 weights (avoid automatic f16 casting on device).
3. Quantize on CPU from an f32 CPU tensor instead.
4. Check tensor.dtype() == DType::F32 before calling quantize.

Example fix

// before
qstorage.quantize(&f16_cuda_storage)?;
// after
let f32_storage = tensor.to_dtype(candle_core::DType::F32)?.to_device(&Device::Cuda)?;
qstorage.quantize(&f32_storage)?;

if tensor.dtype() != candle_core::DType::F32 {
    let tensor = tensor.to_dtype(candle_core::DType::F32)?;
}
qstorage.quantize(&tensor_to_cuda_f32(&tensor)?)?;

fn is_f32_cuda(t: &candle_core::Tensor) -> bool {
    t.dtype() == candle_core::DType::F32 && t.device().is_cuda()
}

match qstorage.quantize(&src) {
    Err(e) if e.to_string().contains("only f32 can be quantized") => {
        let f32_src = src_tensor.to_dtype(candle_core::DType::F32)?;
        qstorage.quantize(&f32_src)?
    }
    r => r?,
}

When it happens

Trigger: Calling quantize on a CudaQuantizedStorage whose source storage slice is not CudaStorageSlice::F32 — e.g. passing a half-precision (f16/bf16) CUDA tensor to quantize, or a tensor produced by an op that yielded another dtype.

Common situations: Quantizing a model whose weights were cast to f16 for CUDA inference; pipeline code that assumes all weights are f32 when they were converted to half; copying quantization code from a CPU example to a CUDA tensor stored in f16.

Related errors

• The given quantized dtype {:?} is not supported for indexed_
• quantized embedding requires contiguous ids
• quantized embedding hidden size {hidden} is not divisible by
• quantized tensor has {} bytes, expected {expected_size}
• unexpected rhs shape in dmmv {:?}

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/8b7f689c8b2bef2b. Report an issue: GitHub.