What causes "rmsnorm is not implemented for {dt1:?} {dt2:?}" in huggingface/candle?

Cast both x and alpha to the same supported dtype (F32) before rms_norm

rmsnorm is not implemented for {dt1:?} {dt2:?}

Error message

rmsnorm is not implemented for {dt1:?} {dt2:?}

What it means

The Metal RMSNorm op dispatches to shaders rmsnorm_f32/f16/bf16, and both the input and alpha must have the same supported dtype. Any other combination (F64, integers, or mismatched dtypes) has no kernel and bails with this message naming both dtypes.

Source

Thrown at candle-nn/src/ops.rs:627

    #[cfg(feature = "metal")]
    fn metal_fwd(
        &self,
        s1: &candle::MetalStorage,
        l1: &Layout,
        s2: &candle::MetalStorage,
        l2: &Layout,
    ) -> Result<(candle::MetalStorage, Shape)> {
        use candle::backend::BackendStorage;
        let device = s1.device();
        let encoder = device.command_encoder()?;
        encoder.set_label("rmsnorm");
        let kernels = device.kernels();
        let name = match (s1.dtype(), s2.dtype()) {
            (DType::F32, DType::F32) => "rmsnorm_f32",
            (DType::F16, DType::F16) => "rmsnorm_f16",
            (DType::BF16, DType::BF16) => "rmsnorm_bf16",
            (dt1, dt2) => candle::bail!("rmsnorm is not implemented for {dt1:?} {dt2:?}"),
        };

        if !(l1.is_contiguous() && l2.is_contiguous()) {
            candle::bail!("Non contiguous rmsnorm is not implemented");
        }

        let last_dim = l1.dims()[l1.shape().rank() - 1];
        let elem_count = l1.shape().elem_count();
        let output = device
            .new_buffer_builder()
            .with_size_for(elem_count, s1.dtype())
            .with_label("rmsnorm")
            .build()?;
        candle_metal_kernels::call_rms_norm(
            device.metal_device(),
            &encoder,
            kernels,
            name,

View on GitHub (pinned to d5fee525bf)

Solutions

Cast both x and alpha to the same supported dtype (F32) before rms_norm
Audit checkpoint loading so norm weights are converted with the model's compute dtype
Keep the whole model on one dtype per device to avoid mixed-dtype norm layers

Example fix

// before
let out = rms_norm(&x, &alpha, eps)?; // x: F32, alpha: BF16
// after
let alpha = alpha.to_dtype(x.dtype())?;
let out = rms_norm(&x, &alpha, eps)?;

Defensive patterns

Strategy: type-guard

Validate before calling

fn ensure_metal_rmsnorm(x: &Tensor, alpha: &Tensor) -> Result<()> {
    if x.dtype() != alpha.dtype() || !matches!(x.dtype(), DType::F32 | DType::F16 | DType::BF16) {
        bail!("Metal rmsnorm needs matching F32/F16/BF16 dtypes, got {:?}/{:?}", x.dtype(), alpha.dtype());
    }
    Ok(())
}

Type guard

fn metal_rmsnorm_ok(x: &Tensor, a: &Tensor) -> bool {
    x.dtype() == a.dtype() && matches!(x.dtype(), DType::F32 | DType::F16 | DType::BF16)
}

Try / catch

let alpha = if metal_rmsnorm_ok(&x, &alpha) { alpha } else { alpha.to_dtype(x.dtype())? };

Prevention

Cast norm weights to the model compute dtype at load
Avoid F64 entirely on Metal
Enforce one dtype per model/device; add dtype asserts in layer init

When it happens

Trigger: Calling ops::rms_norm on a Metal tensor pair where (x.dtype, alpha.dtype) is not exactly (F32,F32), (F16,F16), or (BF16,BF16) — e.g. F64 input, or F32 x with BF16 alpha.

Common situations: F64 tensors on Apple GPU (unsupported); mixed-precision checkpoints where scales are F16 but activations F32; dtype cast applied to activations but not norm weights.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/2d7f55286de699cd. Report an issue: GitHub.