What causes "unsupported dtype for rmsnorm {:?}" in huggingface/candle?

Cast both input and alpha to the same float dtype (F32 recommended) before rms_norm

unsupported dtype for rmsnorm {:?}

Error message

unsupported dtype for rmsnorm {:?}

What it means

The CPU RMSNorm kernel is generic over BF16, F16, and F32 only. If the two input storages are any other dtype (F64, integers) or mismatched, the match falls through and bails reporting the dtype of the first input.

Source

Thrown at candle-nn/src/ops.rs:537

                }
            } else {
                src.par_chunks(dim_m1)
                    .zip(dst.par_chunks_mut(dim_m1))
                    .for_each(|(src, dst)| {
                        let n = src.len();
                        rms_row(src, alpha, n, eps, dst);
                    });
            }
            let storage = candle::WithDType::to_cpu_storage_owned(dst);
            Ok((storage, Shape::from_dims(dims)))
        }

        use CpuStorage as C;
        match (s1, s2) {
            (C::BF16(s1), C::BF16(s2)) => inner::<half::bf16>(s1, l1, s2, l2, eps),
            (C::F16(s1), C::F16(s2)) => inner::<half::f16>(s1, l1, s2, l2, eps),
            (C::F32(s1), C::F32(s2)) => inner::<f32>(s1, l1, s2, l2, eps),
            _ => candle::bail!("unsupported dtype for rmsnorm {:?}", s1.dtype()),
        }
    }

    #[cfg(feature = "cuda")]
    fn cuda_fwd(
        &self,
        s1: &candle::CudaStorage,
        l1: &Layout,
        s2: &candle::CudaStorage,
        l2: &Layout,
    ) -> Result<(candle::CudaStorage, Shape)> {
        use candle::cuda_backend::cudarc::driver::{
            CudaSlice, DeviceRepr, LaunchConfig, PushKernelArg,
        };
        use candle::cuda_backend::{kernel_name, kernels, Map2, WrapErr};
        use candle::{CudaDevice, WithDType};

        struct S {

View on GitHub (pinned to d5fee525bf)

Solutions

Cast both input and alpha to the same float dtype (F32 recommended) before rms_norm
Ensure alpha dtype matches x dtype exactly (x.to_dtype(alpha.dtype())?)
Avoid F64 for RMSNorm on CPU in candle; downcast to F32

Example fix

// before
let out = rms_norm(&x, &alpha, eps)?; // x: F64, alpha: F64
// after
let out = rms_norm(&x.to_dtype(DType::F32)?, &alpha.to_dtype(DType::F32)?, eps)?;

Defensive patterns

Strategy: type-guard

Validate before calling

fn ensure_rmsnorm_pair(x: &Tensor, alpha: &Tensor) -> Result<()> {
    let ok = matches!((x.dtype(), alpha.dtype()),
        (DType::BF16, DType::BF16) | (DType::F16, DType::F16) | (DType::F32, DType::F32));
    if !ok { bail!("rms_norm needs matching F16/BF16/F32 dtypes, got {:?}/{:?}", x.dtype(), alpha.dtype()); }
    Ok(())
}

Type guard

fn rmsnorm_dtype_ok(x: &Tensor, a: &Tensor) -> bool {
    x.dtype() == a.dtype() && matches!(x.dtype(), DType::F32 | DType::F16 | DType::BF16)
}

Try / catch

let (x, a) = if rmsnorm_dtype_ok(&x, &alpha) { (x, alpha) } else {
    (x.to_dtype(DType::F32)?, alpha.to_dtype(DType::F32)?)
};

Prevention

Keep x and alpha in the same dtype
Avoid F64 for CPU rms_norm; use F32
Cast both together at checkpoint load time

When it happens

Trigger: Calling rms_norm on CPU where x and alpha dtypes are not one of (BF16,BF16), (F16,F16), (F32,F32) — e.g. F64 tensors, or F32 x with F16 alpha.

Common situations: Creating tensors from Rust f64 defaults on CPU; mixing quantized scales (F16) with F32 activations; forgetting to cast after .to_dtype conversions elsewhere.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/2f417b6a792f8aa0. Report an issue: GitHub.