What causes "unsupported dtype for softmax {:?}" in huggingface/candle?

Cast the tensor to a supported float dtype before softmax: t.to_dtype(candle_core::DType::F32)?

unsupported dtype for softmax {:?}

Error message

unsupported dtype for softmax {:?}

What it means

candle-nn's softmax last-dim op only has CPU kernels for BF16, F16, F32, and F64. When the tensor's storage dtype is anything else (e.g. integer or quantized types), the CPU fallback match arm bails with this error instead of computing softmax. Softmax is only mathematically meaningful on float-like dtypes in this library.

Source

Thrown at candle-nn/src/ops.rs:328

                    for (s, d) in src.iter().zip(dst.iter_mut()) {
                        *d = (*s - max).exp();
                    }
                    let mut sum_exp = T::zero();
                    unsafe { T::vec_reduce_sum(dst.as_ptr(), &mut sum_exp, dim_m1) };
                    for d in dst.iter_mut() {
                        *d /= sum_exp
                    }
                });
            let storage = candle::WithDType::to_cpu_storage_owned(dst);
            Ok((storage, Shape::from_dims(dims)))
        }

        match storage {
            CpuStorage::BF16(slice) => softmax::<half::bf16>(slice, layout),
            CpuStorage::F16(slice) => softmax::<half::f16>(slice, layout),
            CpuStorage::F32(slice) => softmax::<f32>(slice, layout),
            CpuStorage::F64(slice) => softmax::<f64>(slice, layout),
            _ => candle::bail!("unsupported dtype for softmax {:?}", storage),
        }
    }

    #[cfg(feature = "cuda")]
    fn cuda_fwd(
        &self,
        storage: &candle::CudaStorage,
        layout: &Layout,
    ) -> Result<(candle::CudaStorage, Shape)> {
        use candle::cuda_backend::cudarc::driver::{
            CudaSlice, DeviceRepr, LaunchConfig, PushKernelArg,
        };
        use candle::cuda_backend::{kernel_name, kernels, Map1, WrapErr};
        use candle::{CudaDevice, WithDType};

        struct S;
        impl Map1 for S {
            fn f<T: DeviceRepr + WithDType>(

View on GitHub (pinned to d5fee525bf)

Solutions

Cast the tensor to a supported float dtype before softmax: t.to_dtype(candle_core::DType::F32)?
Check tensor.dtype() before calling softmax and route unsupported dtypes to a cast or a custom kernel
If you need another dtype, implement a custom UnaryOpT/Map1 kernel for it instead of relying on the built-in softmax

Example fix

// before
let probs = candle_nn::ops::softmax_last_dim(&logits)?; // logits: I64
// after
let logits = logits.to_dtype(DType::F32)?;
let probs = candle_nn::ops::softmax_last_dim(&logits)?;

Defensive patterns

Strategy: validation

Validate before calling

fn ensure_softmax_dtype(t: &candle_core::Tensor) -> candle_core::Result<&candle_core::Tensor> {
    match t.dtype() {
        DType::BF16 | DType::F16 | DType::F32 | DType::F64 => Ok(t),
        _ => candle_core::bail!("softmax needs a float dtype, got {:?}; call .to_dtype(DType::F32)?", t.dtype()),
    }
}

Type guard

fn is_softmax_dtype(d: DType) -> bool {
    matches!(d, DType::BF16 | DType::F16 | DType::F32 | DType::F64)
}

Try / catch

let probs = softmax_last_dim(&logits).map_err(|e| format!("softmax dtype: {e}"))?;

Prevention

Cast to F32 at model boundaries before numeric ops
Check tensor.dtype() in debug assertions before GPU/CPU ops
Avoid integer/quantized dtypes for probability computations

When it happens

Trigger: Calling candle_nn::ops::softmax_last_dim (or softmax over the last dim) on a CPU tensor whose dtype is not BF16/F16/F32/F64, e.g. an I64, U8, F8E4M3 or quantized tensor.

Common situations: Passing an integer tensor of indices or logits cast to the wrong dtype; using an experimental/quantized dtype that only has partial kernel coverage; forgetting to .to_dtype() after loading weights.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/bbec6e45366849e9. Report an issue: GitHub.