{"record":{"id":"ab50375eaad6394f","repo":"huggingface/candle","slug":"expected-cpu-storage-ab5037","errorCode":null,"errorMessage":"Expected CPU storage","messagePattern":"Expected CPU storage","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-transformers/src/models/smol/quantized_smollm3.rs","lineNumber":410,"sourceCode":"        let (b, seq_len, _) = x.dims3()?;\n\n        // Fused decode: raw f32, no tensor ops in hot path.\n        if self.use_flash_attn\n            && x.device().is_cpu()\n            && seq_len == 1\n            && b == 1\n            && x.dtype() == DType::F32\n        {\n            // 1. QKV projections (raw f32 output slices)\n            let q_proj_out = self.q_proj.forward(x)?; // (1, 1, H_q * D)\n            let k_proj_out = self.k_proj.forward(x)?; // (1, 1, H_kv * D)\n            let v_proj_out = self.v_proj.forward(x)?; // (1, 1, H_kv * D)\n\n            // Extract flat f32 slices\n            let (q_g, q_l) = q_proj_out.storage_and_layout();\n            let q_flat: &[f32] = match &*q_g {\n                Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[q_l.start_offset()..],\n                _ => candle::bail!(\"Expected CPU storage\"),\n            };\n            let (k_g, k_l) = k_proj_out.storage_and_layout();\n            let k_flat: &[f32] = match &*k_g {\n                Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[k_l.start_offset()..],\n                _ => candle::bail!(\"Expected CPU storage\"),\n            };\n            let (v_g, v_l) = v_proj_out.storage_and_layout();\n            let v_flat: &[f32] = match &*v_g {\n                Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[v_l.start_offset()..],\n                _ => candle::bail!(\"Expected CPU storage\"),\n            };\n\n            // 2. Copy Q and K into pre-allocated buffers for in-place RoPE (no allocation)\n            let q_len = self.num_heads * self.head_dim;\n            let k_len = self.num_kv_heads * self.head_dim;\n            self.q_rope_buf[..q_len].copy_from_slice(&q_flat[..q_len]);\n            self.k_rope_buf[..k_len].copy_from_slice(&k_flat[..k_len]);\n","sourceCodeStart":392,"sourceCodeEnd":428,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-transformers/src/models/smol/quantized_smollm3.rs#L392-L428","documentation":"The quantized SmolLM3 forward hand-rolls attention by taking raw f32 slices out of the q-projection output via `storage_and_layout()`; this only works for `Storage::Cpu`. If the tensor lives on another device (e.g. CUDA/Metal) storage variant, the flat-slice optimization is impossible and it bails 'Expected CPU storage' at the q_proj site (line 410).","triggerScenarios":"Running quantized_smollm3's forward with the model/device moved to a GPU backend, so q_proj_out's storage is not Cpu — e.g. `.to_device(Device::new_cuda(0))` on the model or inputs.","commonSituations":"Users try to accelerate SmolLM3 by putting it on GPU, but this quantized variant's custom RoPE/attention path is CPU-only; mixing devices so one projection output lands on a different backend.","solutions":["Keep the model and all inputs on the CPU device (`candle_core::Device::Cpu`) when using quantized_smollm3.","Use the non-quantized smollm3 implementation for GPU inference.","Clear any `.to_device(...)` calls on the model, inputs, or KV cache before forward.","If you need GPU support, replace the raw-slice attention block with standard candle ops that dispatch per-device."],"exampleFix":"// before\nlet dev = Device::new_cuda(0)?;\nlet model = QuantizedSmolLM3::load(..., &dev)?;\n// after\nlet dev = Device::Cpu;\nlet model = QuantizedSmolLM3::load(..., &dev)?;","handlingStrategy":"validation","validationCode":"if q_proj_out.device().location() != candle_core::DeviceLocation::Cpu {\n    return Err(\"quantized_smollm3 requires CPU device\".into());\n}","typeGuard":"fn on_cpu(t: &Tensor) -> bool {\n    matches!(t.device(), candle_core::Device::Cpu)\n}","tryCatchPattern":"match model.forward(&xs, pos) {\n    Ok(t) => t,\n    Err(e) if e.to_string().contains(\"Expected CPU storage\") => {\n        candle::bail!(\"quantized_smollm3 is CPU-only; use Device::Cpu or the non-quantized model for GPU\")\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Pin the quantized model, inputs, and KV cache to Device::Cpu.","Use the unquantized smollm3 implementation for CUDA/Metal inference.","Remove or guard .to_device() calls that move tensors off CPU.","Document the CPU-only constraint of this quantized variant in your app config."],"tags":["device","cpu-only","quantization","candle"],"backgroundTag":"unsupported-device-backend","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}