huggingface/candle · error

quantized tensor has {} bytes, expected {expected_size}

Error message

quantized tensor has {} bytes, expected {expected_size}

What it means

After checking hidden divisibility, the embedding path validates that the quantized storage's byte size equals the exact expected size: rows * hidden * type_size / block_size. A mismatch means the storage is truncated, oversized, or was produced for a different shape — i.e. the weight tensor's shape and its quantized buffer disagree, so indexing would read out of bounds.

Source

Thrown at candle-core/src/quantized/cuda.rs:830

    pub fn embedding(
        &self,
        rows: usize,
        hidden: usize,
        ids: &CudaStorage,
        ids_l: &crate::Layout,
    ) -> Result<CudaStorage> {
        if !ids_l.is_contiguous() {
            crate::bail!("quantized embedding requires contiguous ids")
        }
        if !hidden.is_multiple_of(self.dtype.block_size()) {
            crate::bail!(
                "quantized embedding hidden size {hidden} is not divisible by block size {}",
                self.dtype.block_size()
            )
        }
        let expected_size = rows * hidden * self.dtype.type_size() / self.dtype.block_size();
        if self.storage_size_in_bytes() != expected_size {
            crate::bail!(
                "quantized tensor has {} bytes, expected {expected_size}",
                self.storage_size_in_bytes()
            )
        }
        let ids = ids.as_cuda_slice::<u32>()?;
        let ids = match ids_l.contiguous_offsets() {
            Some((o1, o2)) => ids.slice(o1..o2),
            None => Err(crate::Error::RequiresContiguous {
                op: "quantized-embedding",
            }
            .bt())?,
        };
        get_rows(&self.data, self.dtype, hidden, &ids, self.device())
    }

    pub fn fwd(
        &self,
        self_shape: &crate::Shape,

View on GitHub (pinned to d5fee525bf)

Solutions

1. Re-download / regenerate the GGUF model file — the file is likely corrupted or truncated.
2. Verify the rows and hidden arguments match the actual embedding weight shape (vocab_size × dim) in the model config.
3. Do not reshape or slice a quantized tensor; dequantize first, reshape, then re-quantize.
4. Re-quantize the embedding table so storage size matches its shape.

Example fix

// before: rows/hidden from a mismatched config
let out = qtable.embedding(cfg.hidden, &ids)?; // rows=32000, but file was 32768
// after: derive from the weight itself and verify integrity of the model file
let (rows, hidden) = { /* from cfg.vocab_size, cfg.dim matching the gguf */ };

let expected = rows * hidden * qweight.dtype().type_size() / qweight.dtype().block_size();
if qweight.storage_size_in_bytes() != expected {
    anyhow::bail!("corrupt/reshaped quantized embedding: {} bytes, expected {expected}",
        qweight.storage_size_in_bytes());
}

match qweight.embedding(hidden, &ids) {
    Err(e) if e.to_string().contains("bytes, expected") => {
        anyhow::bail!("model file corrupted or reshaped; re-download the GGUF: {e}")
    }
    r => r?,
}

When it happens

Trigger: Calling quantized embedding on a QTensor whose storage_size_in_bytes() does not match rows*hidden derived from the table shape — typically after loading a corrupted/partial GGUF, reshaping a quantized tensor incorrectly, or mismatching rows/hidden arguments with the actual weight.

Common situations: Hand-editing or truncating GGUF files; loading a model checkpoint with a config whose vocab/hidden dims differ from the file; constructing QTensor manually with the wrong dims.

Related errors

• quantized embedding requires contiguous ids
• quantized embedding hidden size {hidden} is not divisible by
• The given quantized dtype {:?} is not supported for indexed_
• only f32 can be quantized
• unexpected rhs shape in dmmv {:?}

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/02c238b13becd55f. Report an issue: GitHub.