huggingface/candle · error

Expected CPU

Error message

Expected CPU

What it means

Later in the same forward, quantized SmolLM3 extracts f32 slices from the (transposed) K tensor to write into the raw KV cache (`write_kv_batch`). It requires `Storage::Cpu` and bails with the shorter message 'Expected CPU' otherwise (line 508/509 site).

Source

Thrown at candle-transformers/src/models/smol/quantized_smollm3.rs:509

        let (q, k) = if self.skip_rope {
            (q, k)
        } else if let Some(rope) = &self.rotary_emb {
            rope.apply_rotary_emb(&q, &k, offset)?
        } else {
            (q, k)
        };

        if self.use_flash_attn && x.device().is_cpu() && b == 1 {
            // Prefill (B=1 only): use InterleavedKvCache + flash_attn
            let kv = self.interleaved_cache.as_mut().unwrap().append(&k, &v)?;
            // Also populate raw cache for subsequent decode steps
            {
                let k_cont = k.squeeze(0)?.transpose(0, 1)?.contiguous()?;
                let v_cont = v.squeeze(0)?.transpose(0, 1)?.contiguous()?;
                let (kg, kl) = k_cont.storage_and_layout();
                let k_data: &[f32] = match &*kg {
                    Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[kl.start_offset()..],
                    _ => candle::bail!("Expected CPU"),
                };
                let (vg, vl) = v_cont.storage_and_layout();
                let v_data: &[f32] = match &*vg {
                    Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[vl.start_offset()..],
                    _ => candle::bail!("Expected CPU"),
                };
                self.raw_cache
                    .as_mut()
                    .unwrap()
                    .write_kv_batch(k_data, v_data, seq_len);
            }

            let scale = 1.0 / (self.head_dim as f32).sqrt();
            let kv_k = kv.narrow(2, 0, self.head_dim)?.unsqueeze(0)?;
            let kv_v = kv.narrow(2, self.head_dim, self.head_dim)?.unsqueeze(0)?;

            let q = q.transpose(1, 2)?.contiguous()?;
            let k = kv_k.contiguous()?;

View on GitHub (pinned to d5fee525bf)

Solutions

1. Keep the whole pipeline (model + KV cache) on `Device::Cpu`.
2. Use the non-quantized smollm3 for GPU inference.
3. Ensure the raw KV cache (`self.raw_cache`) and k/v tensors share the CPU device.
4. Replace the raw cache-write block with candle tensor ops for device generality.

Example fix

// before
let k_data: &[f32] = match &*kg {
    Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[kl.start_offset()..],
    _ => bail!("Expected CPU"),
};
// after: run on CPU
let k_cont = k.squeeze(0)?.transpose(0, 1)?.contiguous()?.to_device(&Device::Cpu)?;
let (kg, kl) = k_cont.storage_and_layout();
let k_data: &[f32] = match &*kg {
    Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[kl.start_offset()..],
    _ => bail!("Expected CPU"),
};

if k_cont.device().location() != candle_core::DeviceLocation::Cpu {
    return Err("KV cache writes require CPU tensors".into());
}

fn on_cpu(t: &Tensor) -> bool {
    matches!(t.device(), candle_core::Device::Cpu)
}

match model.forward(&xs, pos) {
    Ok(t) => t,
    Err(e) if e.to_string().contains("Expected CPU") => {
        candle::bail!("KV-cache path in quantized_smollm3 is CPU-only")
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: After `k.squeeze(0)?.transpose(0, 1)?.contiguous()?`, the resulting k tensor's storage is a non-CPU variant — i.e. the model ran on a GPU device, so the cache write from raw pointers is unsupported.

Common situations: GPU execution attempts of the quantized model; KV cache pre-allocated on a different device than activations; custom backends returning non-Cpu storage.

Related errors

• Expected CPU storage
• kv_cache_enabled=true is not supported
• only kv-repeat = 1 is supported
• cannot copy kv-caches as the transformers have different dep
• empty cache despite pos > 0

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/f3822fabd1560422. Report an issue: GitHub.