huggingface/candle · error

Invalid quantize source storage locations: not on cpu

Error message

Invalid quantize source storage locations: not on cpu

What it means

quantize_onto writes quantized data in place over a CPU source buffer; this variant requires the SOURCE storage to be on CPU (Metal/Cuda destinations quantize_onto a Cpu source). Any other combination, such as a GPU source storage, is rejected with this message.

Source

Thrown at candle-core/src/quantized/mod.rs:207

            (QStorage::Metal(storage), Storage::Metal(src)) => {
                storage.quantize_imatrix(src, imatrix_weights, n_per_row)?
            }
            (QStorage::Cuda(storage), Storage::Cuda(src)) => {
                storage.quantize_imatrix(src, imatrix_weights, n_per_row)?
            }
            _ => crate::bail!("Invalid quantize storage locations do not match"),
        }
        Ok(())
    }

    fn quantize_onto(&mut self, src: &Storage) -> Result<()> {
        match (self, src) {
            (QStorage::Cpu(storage), Storage::Cpu(src)) => {
                storage.from_float(src.as_slice::<f32>()?);
            }
            (QStorage::Metal(storage), Storage::Cpu(src)) => storage.quantize_onto(src)?,
            (QStorage::Cuda(storage), Storage::Cpu(src)) => storage.quantize_onto(src)?,
            _ => crate::bail!("Invalid quantize source storage locations: not on cpu"),
        }
        Ok(())
    }

    fn quantize_imatrix_onto(
        &mut self,
        src: &Storage,
        imatrix_weights: &[f32],
        n_per_row: usize,
    ) -> Result<()> {
        match (self, src) {
            (QStorage::Cpu(storage), Storage::Cpu(src)) => {
                storage.from_float_imatrix(src.as_slice::<f32>()?, imatrix_weights, n_per_row);
            }
            (QStorage::Metal(storage), Storage::Cpu(src)) => {
                storage.quantize_imatrix_onto(src, imatrix_weights, n_per_row)?
            }
            (QStorage::Cuda(storage), Storage::Cpu(src)) => {

View on GitHub (pinned to d5fee525bf)

Solutions

1. Copy the source tensor to CPU (to_device(&Device::Cpu)) before quantize_onto.
2. Or use the device-matched quantize()/quantize_imatrix() variants instead of quantize_onto.
3. Ensure the source is f32 and contiguous on CPU.

Example fix

// before
qstorage.quantize_onto(&gpu_storage)?;
// after
let cpu_storage = gpu_tensor.to_device(&Device::Cpu)?;
qstorage.quantize_onto(cpu_storage.as_slice::<f32>()?)?;

fn ensure_cpu_source(src: &candle_core::Storage) -> candle_core::Result<()> {
    if !matches!(src, candle_core::Storage::Cpu(_)) {
        candle_core::bail!("quantize_onto requires a CPU source storage");
    }
    Ok(())
}

fn is_cpu_storage(s: &candle_core::Storage) -> bool { matches!(s, candle_core::Storage::Cpu(_)) }

When it happens

Trigger: Calling quantize_onto where src is a Metal or Cuda Storage instead of a Cpu f32 storage.

Common situations: Trying to quantize directly from a GPU-resident tensor without first copying it to host memory.

Related errors

• Invalid quantize storage locations do not match
• not implemented
• empty array
• two elements have different len {m} {}
• two elements have different shapes {shape:?} {shape0:?}

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/ad63e1fb00f3f68d. Report an issue: GitHub.