{"record":{"id":"6d303ccf626687f2","repo":"huggingface/candle","slug":"the-accelerate-backend-does-not-support-f16-matmul","errorCode":null,"errorMessage":"the accelerate backend does not support f16 matmul","messagePattern":"the accelerate backend does not support f16 matmul","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-core/src/cpu_backend/mod.rs","lineNumber":1569,"sourceCode":"            (n as i32, b'N')\n        } else if rhs_m1 == k && rhs_m2 == 1 {\n            (k as i32, b'T')\n        } else {\n            Err(self.striding_error(lhs_l, rhs_l, \"non-contiguous rhs\"))?\n        };\n        // The b tensor has dims batching, m, k (lhs)\n        let (ldb, transb) = if (lhs_m1 == 1 || k == 1) && (lhs_m2 == k || m == 1) {\n            (k as i32, b'N')\n        } else if lhs_m1 == m && lhs_m2 == 1 {\n            (m as i32, b'T')\n        } else {\n            Err(self.striding_error(lhs_l, rhs_l, \"non-contiguous lhs\"))?\n        };\n\n        let mut dst = vec![T::zero(); b * m * n];\n        match T::DTYPE {\n            DType::F16 => {\n                crate::bail!(\"the accelerate backend does not support f16 matmul\")\n            }\n            DType::F32 => {\n                for step in 0..b {\n                    let lhs_p = &lhs[step * a_skip..];\n                    let rhs_p = &rhs[step * b_skip..];\n                    let dst_p = &mut dst[step * c_skip..];\n                    unsafe {\n                        let a = rhs_p.as_ptr() as *const f32;\n                        let b = lhs_p.as_ptr() as *const f32;\n                        let c = dst_p.as_mut_ptr() as *mut f32;\n                        let a = std::slice::from_raw_parts(a, a_skip);\n                        let b = std::slice::from_raw_parts(b, b_skip);\n                        let c = std::slice::from_raw_parts_mut(c, c_skip);\n                        crate::accelerate::sgemm(\n                            transa, transb, /* m= */ n as i32, /* n= */ m as i32,\n                            /* k= */ k as i32, /* alpha= */ 1., /* a= */ a,\n                            /* lda= */ lda, /* b= */ b, /* ldb= */ ldb,\n                            /* beta= */ 0., /* c= */ c, /* ldc= */ n as i32,","sourceCodeStart":1551,"sourceCodeEnd":1587,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-core/src/cpu_backend/mod.rs#L1551-L1587","documentation":"candle's CPU matmul path specialized for the accelerate backend only implements f32 (and other float) loops; f16 (half precision) CPU matmul is not implemented, so it bails at runtime. Use of F16 tensors on CPU requires GPU/accelerator backends or explicit casting.","triggerScenarios":"Calling matmul (or ops that lower to it, e.g. linear layers, attention) on a Candle f16 Tensor on the CPU device with the accelerate backend, e.g. Tensor::zeros((2,3), DType::F16, &Device::Cpu)?.matmul(&w,)?.","commonSituations":"Loading quantized/half-precision model weights (common with .half()-style configs) and running inference on CPU; following GPU tutorials verbatim while running on a CPU-only machine.","solutions":["Cast tensors to f32 before the matmul: t.to_dtype(candle_core::DType::F32)?.","Run on a GPU/Metal/CUDA device instead of Device::Cpu, where f16 matmul is supported.","Load model weights in f32 (avoid --dtype f16 / half-precision loaders) when restricted to CPU.","Upcast just before compute and cast back afterwards if memory is a concern."],"exampleFix":"// before\nlet y = x.matmul(&w)?; // x, w are F16 on Device::Cpu\n// after\nlet y = x.to_dtype(DType::F32)?.matmul(&w.to_dtype(DType::F32)?)?.to_dtype(DType::F16)?;","handlingStrategy":"validation","validationCode":"if x.dtype() == DType::F16 && x.device().is_cpu() {\n    return Err(anyhow::anyhow!(\"f16 matmul unsupported on CPU; cast to F32 first\"));\n}","typeGuard":null,"tryCatchPattern":"match result { Err(e) if e.to_string().contains(\"does not support f16 matmul\") => { let r = x.to_dtype(DType::F32)?.matmul(&w.to_dtype(DType::F32)?)?; }, other => other?, }","preventionTips":["On CPU-only deployments, always run in f32; reserve f16 for Metal/CUDA devices.","Check device capabilities before selecting a model dtype in your config loader.","Centralize dtype selection in one config path instead of per-op casts."],"tags":["cpu-backend","dtype","matmul","f16","rust"],"backgroundTag":"unsupported-dtype-backend","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}