{"record":{"id":"8f8eb6056a1787ee","repo":"huggingface/candle","slug":"cannot-copy-kv-caches-as-the-transformers-have-dif","errorCode":null,"errorMessage":"cannot copy kv-caches as the transformers have different depths","messagePattern":"cannot copy kv-caches as the transformers have different depths","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"candle-transformers/src/models/mimi/transformer.rs","lineNumber":648,"sourceCode":"                    .map(|i| 1f32 / theta.powf(i as f32 / (half_dim - 1) as f32))\n                    .collect();\n                let inv_freq_len = inv_freq.len();\n                let inv_freq = Tensor::from_vec(inv_freq, (1, inv_freq_len), dev)?;\n                let freqs = positions.broadcast_mul(&inv_freq)?;\n                let pos_emb =\n                    Tensor::cat(&[freqs.cos()?, freqs.sin()?], D::Minus1)?.to_dtype(xs.dtype())?;\n                xs.broadcast_add(&pos_emb)?\n            }\n        };\n        for layer in self.layers.iter_mut() {\n            xs = layer.forward(&xs, ca_src, mask.as_ref())?;\n        }\n        Ok(xs)\n    }\n\n    pub fn copy_state(&mut self, from: &Self) -> Result<()> {\n        if self.layers.len() != from.layers.len() {\n            candle::bail!(\"cannot copy kv-caches as the transformers have different depths\")\n        }\n        self.layers\n            .iter_mut()\n            .zip(from.layers.iter())\n            .for_each(|(v, w)| v.set_kv_cache(w.self_attn.kv_cache.clone()));\n        Ok(())\n    }\n}\n\nimpl StreamingModule for StreamingTransformer {\n    fn reset_state(&mut self) {\n        self.layers.iter_mut().for_each(|v| v.reset_kv_cache())\n    }\n\n    fn step(&mut self, xs: &StreamTensor) -> Result<StreamTensor> {\n        match xs.as_option() {\n            None => Ok(StreamTensor::empty()),\n            Some(xs) => Ok(StreamTensor::from_tensor(self.forward(xs)?)),","sourceCodeStart":630,"sourceCodeEnd":666,"githubUrl":"https://github.com/huggingface/candle/blob/d5fee525bfde3273eb7c9b75fd2bc4937be867ca/candle-transformers/src/models/mimi/transformer.rs#L630-L666","documentation":"StreamingTransformer::copy_state refuses to copy KV caches when the source and destination transformers have a different number of layers. Cache copying is a zip over layers, so mismatched depths would silently truncate; instead it bails with this message.","triggerScenarios":"Calling copy_state(from) on a StreamingTransformer whose layers.len() differs from from.layers.len(), e.g. resetting a worker with a model built from a different config.","commonSituations":"Multi-worker streaming setups where one worker was constructed with a different mimi depth; hot-swapping models of different sizes; typos in config files leading to different num_layers.","solutions":["Ensure both StreamingTransformer instances are built from the same Config (same layer count)","Rebuild the destination transformer with the source's config before copying","If depths legitimately differ, re-run the input through the new model instead of copying caches"],"exampleFix":"// before\nif dest.layers != src_layers { /* mismatched cfg */ }\ndest.copy_state(&src)?; // bails\n// after\nlet dest = StreamingTransformer::new(&src_cfg, vb_dest)?;\ndest.copy_state(&src)?;","handlingStrategy":"validation","validationCode":"if dest.layers_len() != src.layers_len() {\n    return Err(anyhow::anyhow!(\"transformers must have equal depth to copy kv-caches\"));\n}\ndest.copy_state(&src)?;","typeGuard":"fn same_depth(a: &StreamingTransformer, b: &StreamingTransformer) -> bool {\n    a.layers_len() == b.layers_len()\n}","tryCatchPattern":"if let Err(e) = dest.copy_state(&src) {\n    if e.to_string().contains(\"different depths\") {\n        // rebuild dest with the source config and retry\n        dest = StreamingTransformer::new(&src_cfg, vb_dest)?;\n        dest.copy_state(&src)?;\n    } else { return Err(e.into()); }\n}","preventionTips":["Build all workers/instances from the same Config object or hash-compare configs","Log layer counts when constructing streaming transformers used for cache copy","Avoid hot-swapping models of different sizes into live streaming sessions"],"tags":["rust","candle","mimi","kv-cache","state-mismatch"],"backgroundTag":"kv-cache-state-mismatch","analyzedSha":"d5fee525bfde3273eb7c9b75fd2bc4937be867ca","analyzedAt":"2026-09-02T00:15:47.023Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-09T06:17:21.866Z"}