huggingface/candle · error

quantized embedding hidden size

Error message

quantized embedding hidden size {hidden} is not divisible by block size {}

What it means

Quantized embedding kernels process the hidden dimension in fixed block-size chunks of the quantized format (e.g. 32 for Q8_0/Q4K group quants). If the embedding table's hidden size is not a multiple of the dtype's block size, rows cannot be indexed/dequantized safely, so candle bails with the hidden size and block size in the message.

Solutions

1. Adjust the model hidden size so it is divisible by the quant dtype's block size (e.g. 32 or 256 for K-quants).
2. Choose a quantized dtype whose block size divides the hidden size (e.g. Q8_0 vs a K-quant with different block layout).
3. Keep the embedding layer unquantized (f32/f16) and quantize only other layers.
4. Pad rows to a compatible size — requires re-quantizing the padded table and slicing outputs back.

Example fix

// before (hidden = 100, Q4K block size 256)
let q = qtensor.embedding(h, &ids)?;
// after: pick a config where hidden % block_size == 0, or leave embedding unquantized
let q = weight.to_dtype(candle_core::DType::F32)?.embedding(h, &ids)?;

let block = qweight.dtype().block_size();
assert!(hidden % block == 0,
    "hidden {hidden} not divisible by block size {block}; adjust model config or dtype");
let out = qweight.embedding(h, &ids)?;

match qweight.embedding(hidden, &ids) {
    Err(e) if e.to_string().contains("not divisible by block size") => {
        // fallback: dequantize and use dense embedding
        let w = qweight.dequantize(&device)?;
        w.embedding(&ids)?
    }
    r => r?,
}

When it happens

Trigger: Calling quantized embedding where the number of columns of the embedding weight (hidden) is not divisible by dtype.block_size() — e.g. hidden=100 with a block size of 32.

Common situations: Quantizing a model with an unusual/legacy embedding dimension; changing the model config (dim) without adjusting it to the quant block size; custom architectures with non-standard hidden sizes.

Related errors

• quantized embedding requires contiguous ids
• quantized tensor has
• unexpected rhs shape in dmmv
• unexpected shape for input
• mismatch on matmul dim

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/1726db5f613dff3e. Report an issue: GitHub.

Appendix: source

Thrown at candle-core/src/quantized/cuda.rs:823

        Ok(())
    }

    pub fn storage_size_in_bytes(&self) -> usize {
        self.data.len
    }

    pub fn embedding(
        &self,
        rows: usize,
        hidden: usize,
        ids: &CudaStorage,
        ids_l: &crate::Layout,
    ) -> Result<CudaStorage> {
        if !ids_l.is_contiguous() {
            crate::bail!("quantized embedding requires contiguous ids")
        }
        if !hidden.is_multiple_of(self.dtype.block_size()) {
            crate::bail!(
                "quantized embedding hidden size {hidden} is not divisible by block size {}",
                self.dtype.block_size()
            )
        }
        let expected_size = rows * hidden * self.dtype.type_size() / self.dtype.block_size();
        if self.storage_size_in_bytes() != expected_size {
            crate::bail!(
                "quantized tensor has {} bytes, expected {expected_size}",
                self.storage_size_in_bytes()
            )
        }
        let ids = ids.as_cuda_slice::<u32>()?;
        let ids = match ids_l.contiguous_offsets() {
            Some((o1, o2)) => ids.slice(o1..o2),
            None => Err(crate::Error::RequiresContiguous {
                op: "quantized-embedding",
            }
            .bt())?,

View on GitHub (pinned to d5fee525bf)