cannot copy kv-caches as the transformers have different dep

Error message

cannot copy kv-caches as the transformers have different depths

What it means

StreamingTransformer::copy_state refuses to copy KV caches when the source and destination transformers have a different number of layers. Cache copying is a zip over layers, so mismatched depths would silently truncate; instead it bails with this message.

Source

Thrown at candle-transformers/src/models/mimi/transformer.rs:648

                    .map(|i| 1f32 / theta.powf(i as f32 / (half_dim - 1) as f32))
                    .collect();
                let inv_freq_len = inv_freq.len();
                let inv_freq = Tensor::from_vec(inv_freq, (1, inv_freq_len), dev)?;
                let freqs = positions.broadcast_mul(&inv_freq)?;
                let pos_emb =
                    Tensor::cat(&[freqs.cos()?, freqs.sin()?], D::Minus1)?.to_dtype(xs.dtype())?;
                xs.broadcast_add(&pos_emb)?
            }
        };
        for layer in self.layers.iter_mut() {
            xs = layer.forward(&xs, ca_src, mask.as_ref())?;
        }
        Ok(xs)
    }

    pub fn copy_state(&mut self, from: &Self) -> Result<()> {
        if self.layers.len() != from.layers.len() {
            candle::bail!("cannot copy kv-caches as the transformers have different depths")
        }
        self.layers
            .iter_mut()
            .zip(from.layers.iter())
            .for_each(|(v, w)| v.set_kv_cache(w.self_attn.kv_cache.clone()));
        Ok(())
    }
}

impl StreamingModule for StreamingTransformer {
    fn reset_state(&mut self) {
        self.layers.iter_mut().for_each(|v| v.reset_kv_cache())
    }

    fn step(&mut self, xs: &StreamTensor) -> Result<StreamTensor> {
        match xs.as_option() {
            None => Ok(StreamTensor::empty()),
            Some(xs) => Ok(StreamTensor::from_tensor(self.forward(xs)?)),

View on GitHub (pinned to d5fee525bf)

Solutions

Ensure both StreamingTransformer instances are built from the same Config (same layer count)
Rebuild the destination transformer with the source's config before copying
If depths legitimately differ, re-run the input through the new model instead of copying caches

Example fix

// before
if dest.layers != src_layers { /* mismatched cfg */ }
dest.copy_state(&src)?; // bails
// after
let dest = StreamingTransformer::new(&src_cfg, vb_dest)?;
dest.copy_state(&src)?;

Defensive patterns

Strategy: validation

Validate before calling

if dest.layers_len() != src.layers_len() {
    return Err(anyhow::anyhow!("transformers must have equal depth to copy kv-caches"));
}
dest.copy_state(&src)?;

Type guard

fn same_depth(a: &StreamingTransformer, b: &StreamingTransformer) -> bool {
    a.layers_len() == b.layers_len()
}

Try / catch

if let Err(e) = dest.copy_state(&src) {
    if e.to_string().contains("different depths") {
        // rebuild dest with the source config and retry
        dest = StreamingTransformer::new(&src_cfg, vb_dest)?;
        dest.copy_state(&src)?;
    } else { return Err(e.into()); }
}

Prevention

Build all workers/instances from the same Config object or hash-compare configs
Log layer counts when constructing streaming transformers used for cache copy
Avoid hot-swapping models of different sizes into live streaming sessions

When it happens

Trigger: Calling copy_state(from) on a StreamingTransformer whose layers.len() differs from from.layers.len(), e.g. resetting a worker with a model built from a different config.

Common situations: Multi-worker streaming setups where one worker was constructed with a different mimi depth; hot-swapping models of different sizes; typos in config files leading to different num_layers.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/8f8eb6056a1787ee. Report an issue: GitHub.