What causes "rope_thd is not implemented for {dtype:?}" in huggingface/candle?

Cast the input to F32 (or F16/BF16) before rope on Metal.

rope_thd is not implemented for {dtype:?}

Error message

rope_thd is not implemented for {dtype:?}

What it means

The Metal rope_thd kernel is only compiled for F32, F16 and BF16; the dtype-to-kernel-name match in metal_fwd bails for any other dtype (e.g. F64, which has no metal kernel here, or integer types). This is a Metal-specific capability limitation.

Source

Thrown at candle-nn/src/rotary_emb.rs:792

    ) -> Result<(candle::MetalStorage, Shape)> {
        use candle::backend::BackendStorage;
        let device = src.device();
        let encoder = device.command_encoder()?;
        encoder.set_label("rope_thd");
        let kernels = device.kernels();
        if cos.dtype() != src.dtype() || sin.dtype() != src.dtype() {
            candle::bail!(
                "dtype mismatch in rope {:?} {:?} {:?}",
                src.dtype(),
                cos.dtype(),
                sin.dtype()
            )
        }
        let name = match src.dtype() {
            candle::DType::F32 => "rope_thd_f32",
            candle::DType::F16 => "rope_thd_f16",
            candle::DType::BF16 => "rope_thd_bf16",
            dtype => candle::bail!("rope_thd is not implemented for {dtype:?}"),
        };
        let (b, t, h, d) = l_src.shape().dims4()?;
        let stride_b = if l_cos.dims().len() == 3 && l_sin.dims().len() == 3 {
            h * t * d
        } else {
            0usize
        };
        let el = b * h * t * d;
        let output = device
            .new_buffer_builder()
            .with_size_for(el, src.dtype())
            .with_label("rope_thd")
            .build()?;
        candle_metal_kernels::call_rope_thd(
            device.metal_device(),
            &encoder,
            kernels,
            name,

View on GitHub (pinned to d5fee525bf)

Solutions

Cast the input to F32 (or F16/BF16) before rope on Metal.
Avoid F64 on GPU paths entirely; keep F64 math on CPU and cast results back.
Use xs.to_dtype(DType::F32)? at model boundaries when targeting Metal.
Check device placement: an F64 tensor on a Metal device will always fail here.

Example fix

// before
let xs = Tensor::randn(0f64, 1f64, shape, &metal_dev)?; // f64 on metal
let x = rope.forward(&xs, &cos, &sin)?;
// after
let xs = xs.to_dtype(DType::F32)?;
let cos = cos.to_dtype(DType::F32)?;
let sin = sin.to_dtype(DType::F32)?;
let x = rope.forward(&xs, &cos, &sin)?;

Defensive patterns

Strategy: validation

Validate before calling

if !matches!(xs.dtype(), candle::DType::F32 | candle::DType::F16 | candle::DType::BF16) {
    let xs = xs.to_dtype(candle::DType::F32)?;
}

Type guard

fn metal_supported_float(d: candle::DType) -> bool {
    matches!(d, candle::DType::F32 | candle::DType::F16 | candle::DType::BF16)
}

Try / catch

let out = match rope.forward(&xs, &cos, &sin) {
    Ok(o) => o,
    Err(e) if e.to_string().contains("is not implemented for") => {
        let xs = xs.to_dtype(candle::DType::F32)?;
        rope.forward(&xs, &cos.to_dtype(candle::DType::F32)?, &sin.to_dtype(candle::DType::F32)?)?
    }
    Err(e) => return Err(e),
};

Prevention

Never use F64 tensors on Metal devices; restrict F64 to CPU-only code paths
Cast model inputs to F32/F16/BF16 at the device boundary
Check device placement when a tensor is created (F64 defaults are easy to create on any device)
Pin test tensors to the same dtype used in production inference

When it happens

Trigger: Calling rope_thd on Metal with an F64 tensor (F64 is supported on CPU/CUDA but has no metal kernel), or with integer/byte tensors.

Common situations: Code that uses F64 generically on CPU being moved to an Apple GPU device; test code defaulting to F64; passing position index tensors (I64) by mistake.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/5e578dfd967bcd18. Report an issue: GitHub.