huggingface/candle · error

Invalid quantize storage locations do not match

Error message

Invalid quantize storage locations do not match

What it means

QuantizedStorage::quantize requires the quantized destination storage and the f32 source storage to live on the same device. Any (QStorage, Storage) combination other than Cpu/Cpu, Metal/Metal, or Cuda/Cuda hits this catch-all bail.

Source

Thrown at candle-core/src/quantized/mod.rs:174

        }
    }

    fn size_in_bytes(&self) -> usize {
        match self {
            QStorage::Cpu(storage) => storage.storage_size_in_bytes(),
            QStorage::Metal(storage) => storage.storage_size_in_bytes(),
            QStorage::Cuda(storage) => storage.storage_size_in_bytes(),
        }
    }

    fn quantize(&mut self, src: &Storage) -> Result<()> {
        match (self, src) {
            (QStorage::Cpu(storage), Storage::Cpu(src)) => {
                storage.from_float(src.as_slice::<f32>()?);
            }
            (QStorage::Metal(storage), Storage::Metal(src)) => storage.quantize(src)?,
            (QStorage::Cuda(storage), Storage::Cuda(src)) => storage.quantize(src)?,
            _ => crate::bail!("Invalid quantize storage locations do not match"),
        }
        Ok(())
    }

    fn quantize_imatrix(
        &mut self,
        src: &Storage,
        imatrix_weights: &[f32],
        n_per_row: usize,
    ) -> Result<()> {
        match (self, src) {
            (QStorage::Cpu(storage), Storage::Cpu(src)) => {
                storage.from_float_imatrix(src.as_slice::<f32>()?, imatrix_weights, n_per_row);
            }
            (QStorage::Metal(storage), Storage::Metal(src)) => {
                storage.quantize_imatrix(src, imatrix_weights, n_per_row)?
            }
            (QStorage::Cuda(storage), Storage::Cuda(src)) => {

View on GitHub (pinned to d5fee525bf)

Solutions

1. Move the source tensor to the same device as the quantized storage before quantizing (usually cpu to_device(Device::Cpu)).
2. Create the destination QTensor on the same device as the source.
3. Check tensor.device() on both sides before calling quantize.

Example fix

// before
let q = QTensor::quantize(&tensor_on_metal, GgmlType::Q4K, &Device::Cpu)?;
// after
let tensor = tensor_on_metal.to_device(&Device::Cpu)?;
let q = QTensor::quantize(&tensor, GgmlType::Q4K, &Device::Cpu)?;

fn ensure_same_device(q: &candle_core::quantized::QTensor, src: &candle_core::Tensor) -> candle_core::Result<()> {
    if q.device().location() != src.device().location() {
        candle_core::bail!("quantize: device mismatch {:?} vs {:?}", q.device(), src.device());
    }
    Ok(())
}

fn same_device(a: &candle_core::Device, b: &candle_core::Device) -> bool { a.location() == b.location() }

When it happens

Trigger: Calling quantize on a QTensor whose storage is on a different device than the source tensor (e.g. QStorage::Cpu quantizing a Metal f32 tensor, or vice versa).

Common situations: Quantizing a model checkpoint that was loaded on GPU without first moving tensors to CPU (or matching devices); forgetting .to_device() after changing the default device.

Related errors

• not implemented
• Invalid quantize source storage locations: not on cpu
• input rank ({}) must be >= weight rank ({})
• weight rank ({}) must be <= 4
• seqlens_k must be a cuda tensor

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/6176887a43b7dee6. Report an issue: GitHub.