huggingface/candle · error

not implemented

Error message

not implemented

What it means

QStorage::device_ptr returns the raw device pointer of the underlying buffer, but only the CUDA backend implements it. Calling it on a Metal or Cpu quantized storage hits the unconditional bail 'not implemented'.

Source

Thrown at candle-core/src/quantized/mod.rs:258

    fn data(&self) -> Result<Cow<'_, [u8]>> {
        match self {
            QStorage::Cpu(storage) => {
                let data_ptr = storage.as_ptr();
                let size_in_bytes = storage.storage_size_in_bytes();
                let data = unsafe { std::slice::from_raw_parts(data_ptr, size_in_bytes) };
                Ok(Cow::from(data))
            }
            QStorage::Cuda(storage) => Ok(Cow::from(storage.data()?)),
            QStorage::Metal(storage) => Ok(Cow::from(storage.data()?)),
        }
    }

    pub fn device_ptr(&self) -> Result<*const u8> {
        match self {
            QStorage::Cuda(storage) => storage.device_ptr(),
            QStorage::Metal(_) | QStorage::Cpu(_) => {
                crate::bail!("not implemented");
            }
        }
    }

    #[cfg(feature = "cuda")]
    pub fn device_ptr_with_guard<'a>(
        &'a self,
        stream: &'a crate::cuda_backend::cudarc::driver::CudaStream,
    ) -> Result<(
        *const u8,
        crate::cuda_backend::cudarc::driver::SyncOnDrop<'a>,
    )> {
        match self {
            QStorage::Cuda(storage) => storage.device_ptr_with_guard(stream),
            QStorage::Metal(_) | QStorage::Cpu(_) => {
                crate::bail!("not implemented");
            }
        }

View on GitHub (pinned to d5fee525bf)

Solutions

1. Only call device_ptr on CUDA-resident quantized tensors; check storage device first.
2. Move the tensor to the Cuda device before requesting the pointer (quantize onto Cuda).
3. For CPU access, use dequantize() to obtain f32 data instead of a raw pointer.
4. Use device_ptr_with_guard (also CUDA-only) if you need stream-synchronized access.

Example fix

// before
let ptr = qtensor.storage().device_ptr()?; // fails on Metal/Cpu
// after
if matches!(tensor.device(), Device::Cuda(_)) {
    let ptr = qtensor.storage().device_ptr()?;
} else {
    let f32_storage = qtensor.dequantize(tensor.elem_count())?;
}

fn ensure_cuda_qstorage(s: &candle_core::quantized::QStorage) -> candle_core::Result<()> {
    if !matches!(s, candle_core::quantized::QStorage::Cuda(_)) {
        candle_core::bail!("device_ptr is CUDA-only");
    }
    Ok(())
}

fn is_cuda_qstorage(s: &candle_core::quantized::QStorage) -> bool { matches!(s, candle_core::quantized::QStorage::Cuda(_)) }

match qtensor.storage().device_ptr() {
    Ok(ptr) => use_raw_pointer(ptr),
    Err(e) if e.to_string().contains("not implemented") => {
        // fall back to dequantized CPU access
        let f32_data = qtensor.dequantize(&cpu_device)?;
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling QTensor/QStorage::device_ptr() when the quantized tensor lives on CPU or Metal (typically from custom CUDA-interop code).

Common situations: Passing quantized weights to external CUDA kernels (e.g. llama.cpp style bindings) while the model was loaded on CPU or Metal.

Related errors

• Invalid quantize storage locations do not match
• input rank ({}) must be >= weight rank ({})
• weight rank ({}) must be <= 4
• Invalid quantize source storage locations: not on cpu
• input is not a f32 tensor

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/ae803225a10845c1. Report an issue: GitHub.