huggingface/candle · error

Expected CPU storage

Error message

Expected CPU storage

What it means

The quantized SmolLM3 forward hand-rolls attention by taking raw f32 slices out of the q-projection output via `storage_and_layout()`; this only works for `Storage::Cpu`. If the tensor lives on another device (e.g. CUDA/Metal) storage variant, the flat-slice optimization is impossible and it bails 'Expected CPU storage' at the q_proj site (line 410).

Source

Thrown at candle-transformers/src/models/smol/quantized_smollm3.rs:410

        let (b, seq_len, _) = x.dims3()?;

        // Fused decode: raw f32, no tensor ops in hot path.
        if self.use_flash_attn
            && x.device().is_cpu()
            && seq_len == 1
            && b == 1
            && x.dtype() == DType::F32
        {
            // 1. QKV projections (raw f32 output slices)
            let q_proj_out = self.q_proj.forward(x)?; // (1, 1, H_q * D)
            let k_proj_out = self.k_proj.forward(x)?; // (1, 1, H_kv * D)
            let v_proj_out = self.v_proj.forward(x)?; // (1, 1, H_kv * D)

            // Extract flat f32 slices
            let (q_g, q_l) = q_proj_out.storage_and_layout();
            let q_flat: &[f32] = match &*q_g {
                Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[q_l.start_offset()..],
                _ => candle::bail!("Expected CPU storage"),
            };
            let (k_g, k_l) = k_proj_out.storage_and_layout();
            let k_flat: &[f32] = match &*k_g {
                Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[k_l.start_offset()..],
                _ => candle::bail!("Expected CPU storage"),
            };
            let (v_g, v_l) = v_proj_out.storage_and_layout();
            let v_flat: &[f32] = match &*v_g {
                Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[v_l.start_offset()..],
                _ => candle::bail!("Expected CPU storage"),
            };

            // 2. Copy Q and K into pre-allocated buffers for in-place RoPE (no allocation)
            let q_len = self.num_heads * self.head_dim;
            let k_len = self.num_kv_heads * self.head_dim;
            self.q_rope_buf[..q_len].copy_from_slice(&q_flat[..q_len]);
            self.k_rope_buf[..k_len].copy_from_slice(&k_flat[..k_len]);

View on GitHub (pinned to d5fee525bf)

Solutions

1. Keep the model and all inputs on the CPU device (`candle_core::Device::Cpu`) when using quantized_smollm3.
2. Use the non-quantized smollm3 implementation for GPU inference.
3. Clear any `.to_device(...)` calls on the model, inputs, or KV cache before forward.
4. If you need GPU support, replace the raw-slice attention block with standard candle ops that dispatch per-device.

Example fix

// before
let dev = Device::new_cuda(0)?;
let model = QuantizedSmolLM3::load(..., &dev)?;
// after
let dev = Device::Cpu;
let model = QuantizedSmolLM3::load(..., &dev)?;

if q_proj_out.device().location() != candle_core::DeviceLocation::Cpu {
    return Err("quantized_smollm3 requires CPU device".into());
}

fn on_cpu(t: &Tensor) -> bool {
    matches!(t.device(), candle_core::Device::Cpu)
}

match model.forward(&xs, pos) {
    Ok(t) => t,
    Err(e) if e.to_string().contains("Expected CPU storage") => {
        candle::bail!("quantized_smollm3 is CPU-only; use Device::Cpu or the non-quantized model for GPU")
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Running quantized_smollm3's forward with the model/device moved to a GPU backend, so q_proj_out's storage is not Cpu — e.g. `.to_device(Device::new_cuda(0))` on the model or inputs.

Common situations: Users try to accelerate SmolLM3 by putting it on GPU, but this quantized variant's custom RoPE/attention path is CPU-only; mixing devices so one projection output lands on a different backend.

Related errors

• Expected CPU
• {} is a dummy type and cannot be constructed
• {} is a dummy type and cannot be converted
• {} is a dummy type and cannot be converted to scalar
• {} is a dummy type and does not support storage

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/ab50375eaad6394f. Report an issue: GitHub.