huggingface/candle · error

Expected f32/f16

Error message

Expected f32/f16

What it means

This error is thrown by the quantized matmul fast-path ( QuantizedMatMul::cpu_fwd ) when the LHS activation tensor storage is neither F32 nor F16. The quantized CPU kernels (gemmv/gemm for QKK types) are only implemented for f32 and f16 activations, so any other dtype (e.g. bf16, u8, i64) reaching this op on CPU bails.

Solutions

1. Convert the activation tensor to DType::F32 before the quantized matmul: tensor.to_dtype(DType::F32)?
2. Alternatively convert to DType::F16 if the surrounding model uses half precision
3. Check where the tensor dtype changed (e.g. a .to_dtype call or model config dtype) and keep activations f32/f16 for quantized layers

Example fix

// before
let x = x.to_dtype(DType::BF16)?;
let y = qmatmul.forward(&x)?;
// after
let x = x.to_dtype(DType::F32)?;
let y = qmatmul.forward(&x)?;

if !matches!(x.dtype(), candle::DType::F32 | candle::DType::F16) {
    x = x.to_dtype(candle::DType::F32)?;
}
let y = qmatmul.forward(&x)?;

fn is_quantized_mm_ok(t: &candle_core::Tensor) -> bool {
    matches!(t.dtype(), candle_core::DType::F32 | candle_core::DType::F16)
}

let y = qmatmul.forward(&x).or_else(|_| qmatmul.forward(&x.to_dtype(candle::DType::F32)?))?;

When it happens

Trigger: Calling QMatMul::forward / Tensor::quantized_matmul on CPU with a left-hand tensor whose dtype is not F32 or F16, e.g. after .to_dtype(DType::BF16) or passing an integer/quantized tensor as the activation input.

Common situations: Loading a GGUF model and casting activations to bf16 for memory savings; accidentally feeding a quantized or integer tensor as the hidden-state input; mixing a dtype conversion (bf16) with quantized inference in candle.

Related errors

• attribute of type TENSOR has an unsupported data_type
• attribute of type TENSOR was an invalid data_type number
• different inner dimensions in broadcast matmul
• input rank ( ) must be >= weight rank ( )
• metal col2im1d not implemented

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/7962ddc3b9a51a08. Report an issue: GitHub.

Appendix: source

Thrown at candle-core/src/quantized/mod.rs:991

                    (dst_shape.elem_count() / n, k, n),
                    slice,
                    &mut dst_storage,
                )?;
                Ok((crate::CpuStorage::F32(dst_storage), dst_shape))
            }
            DType::F16 => {
                let slice = storage.as_slice::<f16>()?;
                let slice =
                    &slice[layout.start_offset()..layout.start_offset() + src_shape.elem_count()];
                let mut dst_storage = vec![f16::ZERO; dst_shape.elem_count()];
                self_storage.matmul_t_f16(
                    (dst_shape.elem_count() / n, k, n),
                    slice,
                    &mut dst_storage,
                )?;
                Ok((crate::CpuStorage::F16(dst_storage), dst_shape))
            }
            _ => crate::bail!("Expected f32/f16"),
        }
    }

    fn metal_fwd(
        &self,
        storage: &crate::MetalStorage,
        layout: &crate::Layout,
    ) -> Result<(crate::MetalStorage, Shape)> {
        let self_storage = match &self.storage {
            QStorage::Metal(metal) => metal,
            _ => unreachable!("Cannot call metal matmul on non metal QTensor"),
        };
        self_storage.fwd(&self.shape, storage, layout)
    }

    fn cuda_fwd(
        &self,
        storage: &crate::CudaStorage,

View on GitHub (pinned to d5fee525bf)