huggingface/candle · error

quantized embedding hidden size {hidden} is not divisible by

Error message

quantized embedding hidden size {hidden} is not divisible by block size {}

What it means

Quantized embedding kernels process the hidden dimension in fixed block-size chunks of the quantized format (e.g. 32 for Q8_0/Q4K group quants). If the embedding table's hidden size is not a multiple of the dtype's block size, rows cannot be indexed/dequantized safely, so candle bails with the hidden size and block size in the message.

Source

Thrown at candle-core/src/quantized/cuda.rs:823

        Ok(())
    }

    pub fn storage_size_in_bytes(&self) -> usize {
        self.data.len
    }

    pub fn embedding(
        &self,
        rows: usize,
        hidden: usize,
        ids: &CudaStorage,
        ids_l: &crate::Layout,
    ) -> Result<CudaStorage> {
        if !ids_l.is_contiguous() {
            crate::bail!("quantized embedding requires contiguous ids")
        }
        if !hidden.is_multiple_of(self.dtype.block_size()) {
            crate::bail!(
                "quantized embedding hidden size {hidden} is not divisible by block size {}",
                self.dtype.block_size()
            )
        }
        let expected_size = rows * hidden * self.dtype.type_size() / self.dtype.block_size();
        if self.storage_size_in_bytes() != expected_size {
            crate::bail!(
                "quantized tensor has {} bytes, expected {expected_size}",
                self.storage_size_in_bytes()
            )
        }
        let ids = ids.as_cuda_slice::<u32>()?;
        let ids = match ids_l.contiguous_offsets() {
            Some((o1, o2)) => ids.slice(o1..o2),
            None => Err(crate::Error::RequiresContiguous {
                op: "quantized-embedding",
            }
            .bt())?,

View on GitHub (pinned to d5fee525bf)

Solutions

1. Adjust the model hidden size so it is divisible by the quant dtype's block size (e.g. 32 or 256 for K-quants).
2. Choose a quantized dtype whose block size divides the hidden size (e.g. Q8_0 vs a K-quant with different block layout).
3. Keep the embedding layer unquantized (f32/f16) and quantize only other layers.
4. Pad rows to a compatible size — requires re-quantizing the padded table and slicing outputs back.

Example fix

// before (hidden = 100, Q4K block size 256)
let q = qtensor.embedding(h, &ids)?;
// after: pick a config where hidden % block_size == 0, or leave embedding unquantized
let q = weight.to_dtype(candle_core::DType::F32)?.embedding(h, &ids)?;

let block = qweight.dtype().block_size();
assert!(hidden % block == 0,
    "hidden {hidden} not divisible by block size {block}; adjust model config or dtype");
let out = qweight.embedding(h, &ids)?;

match qweight.embedding(hidden, &ids) {
    Err(e) if e.to_string().contains("not divisible by block size") => {
        // fallback: dequantize and use dense embedding
        let w = qweight.dequantize(&device)?;
        w.embedding(&ids)?
    }
    r => r?,
}

When it happens

Trigger: Calling quantized embedding where the number of columns of the embedding weight (hidden) is not divisible by dtype.block_size() — e.g. hidden=100 with a block size of 32.

Common situations: Quantizing a model with an unusual/legacy embedding dimension; changing the model config (dim) without adjusting it to the quant block size; custom architectures with non-standard hidden sizes.

Related errors

• quantized embedding requires contiguous ids
• quantized tensor has {} bytes, expected {expected_size}
• unexpected rhs shape in dmmv {:?}
• unexpected shape for input {s:?}
• The given quantized dtype {:?} is not supported for indexed_

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/1726db5f613dff3e. Report an issue: GitHub.