huggingface/candle · error

Expected f32/f16

Error message

Expected f32/f16

What it means

This error is thrown by the quantized matmul fast-path ( QuantizedMatMul::cpu_fwd ) when the LHS activation tensor storage is neither F32 nor F16. The quantized CPU kernels (gemmv/gemm for QKK types) are only implemented for f32 and f16 activations, so any other dtype (e.g. bf16, u8, i64) reaching this op on CPU bails.

Source

Thrown at candle-core/src/quantized/mod.rs:991

                    (dst_shape.elem_count() / n, k, n),
                    slice,
                    &mut dst_storage,
                )?;
                Ok((crate::CpuStorage::F32(dst_storage), dst_shape))
            }
            DType::F16 => {
                let slice = storage.as_slice::<f16>()?;
                let slice =
                    &slice[layout.start_offset()..layout.start_offset() + src_shape.elem_count()];
                let mut dst_storage = vec![f16::ZERO; dst_shape.elem_count()];
                self_storage.matmul_t_f16(
                    (dst_shape.elem_count() / n, k, n),
                    slice,
                    &mut dst_storage,
                )?;
                Ok((crate::CpuStorage::F16(dst_storage), dst_shape))
            }
            _ => crate::bail!("Expected f32/f16"),
        }
    }

    fn metal_fwd(
        &self,
        storage: &crate::MetalStorage,
        layout: &crate::Layout,
    ) -> Result<(crate::MetalStorage, Shape)> {
        let self_storage = match &self.storage {
            QStorage::Metal(metal) => metal,
            _ => unreachable!("Cannot call metal matmul on non metal QTensor"),
        };
        self_storage.fwd(&self.shape, storage, layout)
    }

    fn cuda_fwd(
        &self,
        storage: &crate::CudaStorage,

View on GitHub (pinned to d5fee525bf)

Solutions

1. Convert the activation tensor to DType::F32 before the quantized matmul: tensor.to_dtype(DType::F32)?
2. Alternatively convert to DType::F16 if the surrounding model uses half precision
3. Check where the tensor dtype changed (e.g. a .to_dtype call or model config dtype) and keep activations f32/f16 for quantized layers

Example fix

// before
let x = x.to_dtype(DType::BF16)?;
let y = qmatmul.forward(&x)?;
// after
let x = x.to_dtype(DType::F32)?;
let y = qmatmul.forward(&x)?;

if !matches!(x.dtype(), candle::DType::F32 | candle::DType::F16) {
    x = x.to_dtype(candle::DType::F32)?;
}
let y = qmatmul.forward(&x)?;

fn is_quantized_mm_ok(t: &candle_core::Tensor) -> bool {
    matches!(t.dtype(), candle_core::DType::F32 | candle_core::DType::F16)
}

let y = qmatmul.forward(&x).or_else(|_| qmatmul.forward(&x.to_dtype(candle::DType::F32)?))?;

When it happens

Trigger: Calling QMatMul::forward / Tensor::quantized_matmul on CPU with a left-hand tensor whose dtype is not F32 or F16, e.g. after .to_dtype(DType::BF16) or passing an integer/quantized tensor as the activation input.

Common situations: Loading a GGUF model and casting activations to bf16 for memory savings; accidentally feeding a quantized or integer tensor as the hidden-state input; mixing a dtype conversion (bf16) with quantized inference in candle.

Related errors

• the accelerate backend does not support f16 matmul
• Metal contiguous unary {name} {dtype:?} not implemented
• Metal strided unary {name} {dtype:?} not implemented
• Metal where_cond {left:?} {right:?} not implemented
• Metal conv1d {dtype:?} not implemented

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/7962ddc3b9a51a08. Report an issue: GitHub.