huggingface/candle · error

SDPA has no cpu impl

Error message

SDPA has no cpu impl

What it means

The Sdpa custom op in candle-nn is implemented only for the Metal backend; its cpu_fwd unconditionally bails. Scaled-dot-product attention via this fused op therefore cannot run when tensors are on the CPU device.

Source

Thrown at candle-nn/src/ops.rs:1029

    mask: Option<Tensor>,
    do_causal: bool,
}

impl candle::CustomOp3 for Sdpa {
    fn name(&self) -> &'static str {
        "metal-sdpa"
    }

    fn cpu_fwd(
        &self,
        _s1: &CpuStorage,
        _l1: &Layout,
        _s2: &CpuStorage,
        _l2: &Layout,
        _s3: &CpuStorage,
        _l3: &Layout,
    ) -> Result<(CpuStorage, Shape)> {
        candle::bail!("SDPA has no cpu impl")
    }

    #[cfg(feature = "metal")]
    fn metal_fwd(
        &self,
        q: &candle::MetalStorage,
        q_l: &Layout,
        k: &candle::MetalStorage,
        k_l: &Layout,
        v: &candle::MetalStorage,
        v_l: &Layout,
    ) -> Result<(candle::MetalStorage, Shape)> {
        use candle::backend::BackendStorage;
        use candle_metal_kernels::SdpaDType;

        let device = q.device();

        let out_dims = vec![q_l.dim(0)?, q_l.dim(1)?, q_l.dim(2)?, v_l.dim(3)?];

View on GitHub (pinned to d5fee525bf)

Solutions

1. Run on a Metal device (Device::new_metal) so metal_fwd is used
2. Replace the fused SDPA call with a manual attention implementation using standard Tensor ops (matmul, softmax) that work on CPU
3. Use a different backend build of candle (cuda/cudarc) if available, with its own SDPA implementation
4. Guard code paths so the fused SDPA is only reached when the device is Metal

Example fix

// before
let out = sdpa(&q, &k, &v, &mask, false, Some(1.0))?; // on CPU device
// after
let dev = candle::Device::new_metal(0)?;
let q = q.to_device(&dev)?; // k, v, mask likewise
let out = sdpa(&q, &k, &v, &mask, false, Some(1.0))?;

fn sdpa_device_ok(dev: &candle::Device) -> bool {
    matches!(dev, candle::Device::Metal(_))
}

fn is_metal(dev: &candle::Device) -> bool { matches!(dev, candle::Device::Metal(_)) }

match try_fused_sdpa(&q, &k, &v) {
    Ok(y) => y,
    Err(e) if e.to_string().contains("no cpu impl") => manual_attention(&q, &k, &v, scale)?,
    Err(e) => return Err(e),
}

When it happens

Trigger: Invoking the fused SDPA op (e.g. via sdpa helper in candle-nn) with CPU tensors, or running a model on CPU that hits the Sdpa CustomOp3 path.

Common situations: Running tests or inference on a Linux/Windows machine or explicitly with Device::Cpu; CI environments without Apple Silicon; debugging Metal code with a CPU device.

Related errors

• input has to be contiguous
• not implemented yet, self.device: {:?}, device: {:?}
• unsupported dtype for softmax {:?}
• alpha has to be contiguous
• unsupported dtype for rmsnorm {:?}

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/3331db8a0de97d62. Report an issue: GitHub.