{"record":{"id":"bbec6e45366849e9","repo":"huggingface/candle","slug":"unsupported-dtype-for-softmax","errorCode":null,"errorMessage":"unsupported dtype for softmax {:?}","messagePattern":"unsupported dtype for softmax (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-nn/src/ops.rs","lineNumber":328,"sourceCode":"                    for (s, d) in src.iter().zip(dst.iter_mut()) {\n                        *d = (*s - max).exp();\n                    }\n                    let mut sum_exp = T::zero();\n                    unsafe { T::vec_reduce_sum(dst.as_ptr(), &mut sum_exp, dim_m1) };\n                    for d in dst.iter_mut() {\n                        *d /= sum_exp\n                    }\n                });\n            let storage = candle::WithDType::to_cpu_storage_owned(dst);\n            Ok((storage, Shape::from_dims(dims)))\n        }\n\n        match storage {\n            CpuStorage::BF16(slice) => softmax::<half::bf16>(slice, layout),\n            CpuStorage::F16(slice) => softmax::<half::f16>(slice, layout),\n            CpuStorage::F32(slice) => softmax::<f32>(slice, layout),\n            CpuStorage::F64(slice) => softmax::<f64>(slice, layout),\n            _ => candle::bail!(\"unsupported dtype for softmax {:?}\", storage),\n        }\n    }\n\n    #[cfg(feature = \"cuda\")]\n    fn cuda_fwd(\n        &self,\n        storage: &candle::CudaStorage,\n        layout: &Layout,\n    ) -> Result<(candle::CudaStorage, Shape)> {\n        use candle::cuda_backend::cudarc::driver::{\n            CudaSlice, DeviceRepr, LaunchConfig, PushKernelArg,\n        };\n        use candle::cuda_backend::{kernel_name, kernels, Map1, WrapErr};\n        use candle::{CudaDevice, WithDType};\n\n        struct S;\n        impl Map1 for S {\n            fn f<T: DeviceRepr + WithDType>(","sourceCodeStart":310,"sourceCodeEnd":346,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-nn/src/ops.rs#L310-L346","documentation":"candle-nn's softmax last-dim op only has CPU kernels for BF16, F16, F32, and F64. When the tensor's storage dtype is anything else (e.g. integer or quantized types), the CPU fallback match arm bails with this error instead of computing softmax. Softmax is only mathematically meaningful on float-like dtypes in this library.","triggerScenarios":"Calling candle_nn::ops::softmax_last_dim (or softmax over the last dim) on a CPU tensor whose dtype is not BF16/F16/F32/F64, e.g. an I64, U8, F8E4M3 or quantized tensor.","commonSituations":"Passing an integer tensor of indices or logits cast to the wrong dtype; using an experimental/quantized dtype that only has partial kernel coverage; forgetting to .to_dtype() after loading weights.","solutions":["Cast the tensor to a supported float dtype before softmax: t.to_dtype(candle_core::DType::F32)?","Check tensor.dtype() before calling softmax and route unsupported dtypes to a cast or a custom kernel","If you need another dtype, implement a custom UnaryOpT/Map1 kernel for it instead of relying on the built-in softmax"],"exampleFix":"// before\nlet probs = candle_nn::ops::softmax_last_dim(&logits)?; // logits: I64\n// after\nlet logits = logits.to_dtype(DType::F32)?;\nlet probs = candle_nn::ops::softmax_last_dim(&logits)?;","handlingStrategy":"validation","validationCode":"fn ensure_softmax_dtype(t: &candle_core::Tensor) -> candle_core::Result<&candle_core::Tensor> {\n    match t.dtype() {\n        DType::BF16 | DType::F16 | DType::F32 | DType::F64 => Ok(t),\n        _ => candle_core::bail!(\"softmax needs a float dtype, got {:?}; call .to_dtype(DType::F32)?\", t.dtype()),\n    }\n}","typeGuard":"fn is_softmax_dtype(d: DType) -> bool {\n    matches!(d, DType::BF16 | DType::F16 | DType::F32 | DType::F64)\n}","tryCatchPattern":"let probs = softmax_last_dim(&logits).map_err(|e| format!(\"softmax dtype: {e}\"))?;","preventionTips":["Cast to F32 at model boundaries before numeric ops","Check tensor.dtype() in debug assertions before GPU/CPU ops","Avoid integer/quantized dtypes for probability computations"],"tags":["dtype","softmax","cpu","candle"],"backgroundTag":"unsupported-dtype-for-op","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}