{"record":{"id":"f3822fabd1560422","repo":"huggingface/candle","slug":"expected-cpu-f3822f","errorCode":null,"errorMessage":"Expected CPU","messagePattern":"Expected CPU","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-transformers/src/models/smol/quantized_smollm3.rs","lineNumber":509,"sourceCode":"        let (q, k) = if self.skip_rope {\n            (q, k)\n        } else if let Some(rope) = &self.rotary_emb {\n            rope.apply_rotary_emb(&q, &k, offset)?\n        } else {\n            (q, k)\n        };\n\n        if self.use_flash_attn && x.device().is_cpu() && b == 1 {\n            // Prefill (B=1 only): use InterleavedKvCache + flash_attn\n            let kv = self.interleaved_cache.as_mut().unwrap().append(&k, &v)?;\n            // Also populate raw cache for subsequent decode steps\n            {\n                let k_cont = k.squeeze(0)?.transpose(0, 1)?.contiguous()?;\n                let v_cont = v.squeeze(0)?.transpose(0, 1)?.contiguous()?;\n                let (kg, kl) = k_cont.storage_and_layout();\n                let k_data: &[f32] = match &*kg {\n                    Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[kl.start_offset()..],\n                    _ => candle::bail!(\"Expected CPU\"),\n                };\n                let (vg, vl) = v_cont.storage_and_layout();\n                let v_data: &[f32] = match &*vg {\n                    Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[vl.start_offset()..],\n                    _ => candle::bail!(\"Expected CPU\"),\n                };\n                self.raw_cache\n                    .as_mut()\n                    .unwrap()\n                    .write_kv_batch(k_data, v_data, seq_len);\n            }\n\n            let scale = 1.0 / (self.head_dim as f32).sqrt();\n            let kv_k = kv.narrow(2, 0, self.head_dim)?.unsqueeze(0)?;\n            let kv_v = kv.narrow(2, self.head_dim, self.head_dim)?.unsqueeze(0)?;\n\n            let q = q.transpose(1, 2)?.contiguous()?;\n            let k = kv_k.contiguous()?;","sourceCodeStart":491,"sourceCodeEnd":527,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-transformers/src/models/smol/quantized_smollm3.rs#L491-L527","documentation":"Later in the same forward, quantized SmolLM3 extracts f32 slices from the (transposed) K tensor to write into the raw KV cache (`write_kv_batch`). It requires `Storage::Cpu` and bails with the shorter message 'Expected CPU' otherwise (line 508/509 site).","triggerScenarios":"After `k.squeeze(0)?.transpose(0, 1)?.contiguous()?`, the resulting k tensor's storage is a non-CPU variant — i.e. the model ran on a GPU device, so the cache write from raw pointers is unsupported.","commonSituations":"GPU execution attempts of the quantized model; KV cache pre-allocated on a different device than activations; custom backends returning non-Cpu storage.","solutions":["Keep the whole pipeline (model + KV cache) on `Device::Cpu`.","Use the non-quantized smollm3 for GPU inference.","Ensure the raw KV cache (`self.raw_cache`) and k/v tensors share the CPU device.","Replace the raw cache-write block with candle tensor ops for device generality."],"exampleFix":"// before\nlet k_data: &[f32] = match &*kg {\n    Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[kl.start_offset()..],\n    _ => bail!(\"Expected CPU\"),\n};\n// after: run on CPU\nlet k_cont = k.squeeze(0)?.transpose(0, 1)?.contiguous()?.to_device(&Device::Cpu)?;\nlet (kg, kl) = k_cont.storage_and_layout();\nlet k_data: &[f32] = match &*kg {\n    Storage::Cpu(cpu) => &cpu.as_slice::<f32>()?[kl.start_offset()..],\n    _ => bail!(\"Expected CPU\"),\n};","handlingStrategy":"validation","validationCode":"if k_cont.device().location() != candle_core::DeviceLocation::Cpu {\n    return Err(\"KV cache writes require CPU tensors\".into());\n}","typeGuard":"fn on_cpu(t: &Tensor) -> bool {\n    matches!(t.device(), candle_core::Device::Cpu)\n}","tryCatchPattern":"match model.forward(&xs, pos) {\n    Ok(t) => t,\n    Err(e) if e.to_string().contains(\"Expected CPU\") => {\n        candle::bail!(\"KV-cache path in quantized_smollm3 is CPU-only\")\n    }\n    Err(e) => return Err(e.into()),\n}","preventionTips":["Allocate the raw KV cache on Device::Cpu and keep k/v there too.","Verify cache and activations share a device before write_kv_batch.","Choose the non-quantized model for GPU serving.","Add a device check wrapper around cache writes in debug builds."],"tags":["device","cpu-only","kv-cache","candle"],"backgroundTag":"unsupported-device-backend","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}