huggingface/candle · error

the accelerate backend does not support f16 matmul

Error message

the accelerate backend does not support f16 matmul

What it means

candle's CPU matmul path specialized for the accelerate backend only implements f32 (and other float) loops; f16 (half precision) CPU matmul is not implemented, so it bails at runtime. Use of F16 tensors on CPU requires GPU/accelerator backends or explicit casting.

Source

Thrown at candle-core/src/cpu_backend/mod.rs:1569

            (n as i32, b'N')
        } else if rhs_m1 == k && rhs_m2 == 1 {
            (k as i32, b'T')
        } else {
            Err(self.striding_error(lhs_l, rhs_l, "non-contiguous rhs"))?
        };
        // The b tensor has dims batching, m, k (lhs)
        let (ldb, transb) = if (lhs_m1 == 1 || k == 1) && (lhs_m2 == k || m == 1) {
            (k as i32, b'N')
        } else if lhs_m1 == m && lhs_m2 == 1 {
            (m as i32, b'T')
        } else {
            Err(self.striding_error(lhs_l, rhs_l, "non-contiguous lhs"))?
        };

        let mut dst = vec![T::zero(); b * m * n];
        match T::DTYPE {
            DType::F16 => {
                crate::bail!("the accelerate backend does not support f16 matmul")
            }
            DType::F32 => {
                for step in 0..b {
                    let lhs_p = &lhs[step * a_skip..];
                    let rhs_p = &rhs[step * b_skip..];
                    let dst_p = &mut dst[step * c_skip..];
                    unsafe {
                        let a = rhs_p.as_ptr() as *const f32;
                        let b = lhs_p.as_ptr() as *const f32;
                        let c = dst_p.as_mut_ptr() as *mut f32;
                        let a = std::slice::from_raw_parts(a, a_skip);
                        let b = std::slice::from_raw_parts(b, b_skip);
                        let c = std::slice::from_raw_parts_mut(c, c_skip);
                        crate::accelerate::sgemm(
                            transa, transb, /* m= */ n as i32, /* n= */ m as i32,
                            /* k= */ k as i32, /* alpha= */ 1., /* a= */ a,
                            /* lda= */ lda, /* b= */ b, /* ldb= */ ldb,
                            /* beta= */ 0., /* c= */ c, /* ldc= */ n as i32,

View on GitHub (pinned to d5fee525bf)

Solutions

1. Cast tensors to f32 before the matmul: t.to_dtype(candle_core::DType::F32)?.
2. Run on a GPU/Metal/CUDA device instead of Device::Cpu, where f16 matmul is supported.
3. Load model weights in f32 (avoid --dtype f16 / half-precision loaders) when restricted to CPU.
4. Upcast just before compute and cast back afterwards if memory is a concern.

Example fix

// before
let y = x.matmul(&w)?; // x, w are F16 on Device::Cpu
// after
let y = x.to_dtype(DType::F32)?.matmul(&w.to_dtype(DType::F32)?)?.to_dtype(DType::F16)?;

if x.dtype() == DType::F16 && x.device().is_cpu() {
    return Err(anyhow::anyhow!("f16 matmul unsupported on CPU; cast to F32 first"));
}

match result { Err(e) if e.to_string().contains("does not support f16 matmul") => { let r = x.to_dtype(DType::F32)?.matmul(&w.to_dtype(DType::F32)?)?; }, other => other?, }

When it happens

Trigger: Calling matmul (or ops that lower to it, e.g. linear layers, attention) on a Candle f16 Tensor on the CPU device with the accelerate backend, e.g. Tensor::zeros((2,3), DType::F16, &Device::Cpu)?.matmul(&w,)?.

Common situations: Loading quantized/half-precision model weights (common with .half()-style configs) and running inference on CPU; following GPU tutorials verbatim while running on a CPU-only machine.

Related errors

• upcasting is not supported {:?}
• dtype mismatch
• Expected f32/f16
• attribute {} of type TENSOR was an invalid data_type number
• attribute {} of type TENSOR has an unsupported data_type {}

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/6d303ccf626687f2. Report an issue: GitHub.