What causes "input is not a f32 tensor" in huggingface/candle?

Convert the tensor to F32 before applying the custom op: tensor.to_dtype(candle_core::DType::F32)?.

input is not a f32 tensor

Error message

input is not a f32 tensor

What it means

candle's metal_fwd custom op for Metal only supports F32 tensors. Before launching the Metal compute pipeline it checks the storage dtype and bails if it is not DType::F32. Only F32 is implemented so far (a TODO notes more dtypes are planned).

Source

Thrown at candle-core/src/custom_op.rs:748

#[cfg(feature = "ug")]
impl InplaceOp1 for UgIOp1 {
    fn name(&self) -> &'static str {
        self.name
    }

    fn cpu_fwd(&self, _: &mut CpuStorage, _: &Layout) -> Result<()> {
        crate::bail!("ug ops are only supported on metal/cuda at the moment")
    }

    #[cfg(feature = "metal")]
    fn metal_fwd(&self, sto: &mut MetalStorage, layout: &Layout) -> Result<()> {
        use crate::backend::BackendStorage;
        use objc2_metal;

        let elem_count = layout.shape().elem_count();
        if sto.dtype() != crate::DType::F32 {
            // TODO: support more dtypes.
            crate::bail!("input is not a f32 tensor")
        }
        let device = sto.device();
        let encoder = device.command_encoder()?;
        encoder.set_compute_pipeline_state(&self.func);
        candle_metal_kernels::debug_group!(encoder, "{}", self.name);
        let (g, b) = if elem_count.is_multiple_of(32) {
            (elem_count / 32, 32)
        } else {
            (elem_count, 1)
        };
        let grid_dims = objc2_metal::MTLSize {
            width: g,
            height: 1,
            depth: 1,
        };
        let group_dims = candle_metal_kernels::utils::get_block_dims(b, 1, 1);
        let encoder: &candle_metal_kernels::metal::ComputeCommandEncoder = encoder.as_ref();
        encoder.set_output_buffer(0, Some(sto.buffer()), 0);

View on GitHub (pinned to d5fee525bf)

Solutions

Convert the tensor to F32 before applying the custom op: tensor.to_dtype(candle_core::DType::F32)?.
Ensure model weights are loaded/quantized as f32 when targeting Metal custom ops.
If you own the op, extend metal_fwd to support the needed dtype instead of relying on F32 only.

Example fix

// before
let out = tensor.apply(&custom_op)?;
// after
let out = tensor.to_dtype(DType::F32)?.apply(&custom_op)?;

Defensive patterns

Strategy: validation

Validate before calling

if tensor.dtype() != candle_core::DType::F32 {
    tensor = tensor.to_dtype(candle_core::DType::F32)?;
}

Type guard

fn is_f32(t: &candle_core::Tensor) -> bool { t.dtype() == candle_core::DType::F32 }

Prevention

Normalize input tensors to F32 at pipeline entry on Metal devices.
Load weights as f32 when using Metal custom ops.
Check dtype with tensor.dtype() before any custom op.

When it happens

Trigger: Calling Tensor::apply_arisc or any custom op backed by MetalCustomOp when the input tensor storage has dtype other than F32 (e.g. BF16, F16, F64, I64, U8) on a Metal device.

Common situations: Running on Apple Silicon where candle selects the Metal backend; loading a model with bf16/f16 weights; creating tensors with Tensor::new on integer data and applying a custom op.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/24290de87af38d10. Report an issue: GitHub.