huggingface/candle · error

empty cache despite pos > 0

Error message

empty cache despite pos > 0

What it means

Thrown by quantized_recurrent_gemma's forward when the sequence position is past 0 (pos > 0) and incremental decoding is expected, but the internal conv1d_state cache is None. The streaming conv path needs the cached previous inputs to build the full convolution window; without it the model cannot compute the conv output.

Source

Thrown at candle-transformers/src/models/quantized_recurrent_gemma.rs:153

            let x_len = x_branch.dim(D::Minus1)?;
            let pad = self.conv1d_width as i64 - x_len as i64 - 1;
            let padded = match pad.cmp(&0) {
                std::cmp::Ordering::Equal => x_branch.clone(),
                std::cmp::Ordering::Less => {
                    let rev_pad = (-pad) as usize;
                    x_branch.narrow(D::Minus1, rev_pad, x_len - rev_pad)?
                }
                std::cmp::Ordering::Greater => {
                    x_branch.pad_with_zeros(D::Minus1, pad as usize, 0)?
                }
            };
            self.conv1d_state = Some(padded);
            x_branch
                .apply(&self.conv_1d)?
                .narrow(D::Minus1, 0, seq_len)?
        } else {
            let conv_state = match self.conv1d_state.as_ref() {
                None => candle::bail!("empty cache despite pos > 0"),
                Some(s) => Tensor::cat(&[s, &x_branch], D::Minus1)?,
            };
            let w = self.conv_1d.weight().i((.., 0, ..))?;
            let x_branch = conv_state.broadcast_mul(&w)?.sum(D::Minus1)?;
            let x_branch = match self.conv_1d.bias() {
                None => x_branch,
                Some(b) => x_branch.broadcast_add(b)?,
            };
            let x_branch = x_branch.unsqueeze(D::Minus1)?;
            self.conv1d_state = Some(conv_state.i((.., .., 1..))?);
            x_branch
        };
        let x_branch = x_branch.transpose(1, 2)?;
        let x_branch = self.rg_lru.forward(&x_branch, pos)?;
        (x_branch * y_branch)?.apply(&self.linear_out)
    }
}

View on GitHub (pinned to d5fee525bf)

Solutions

  1. Ensure the first forward call starts at pos 0 so the conv1d cache is populated before subsequent pos > 0 calls
  2. Use a single forward code path consistently (don't alternate between cache-seeding and incremental calls)
  3. Create a fresh Model instance if the cache state was invalidated mid-generation

Example fix

// before (starts generation at pos > 0 with empty cache)
let logits = model.forward(&tokens, 5)?;
// after (seed cache from pos 0 first)
let logits = model.forward(&prompt_tokens, 0)?;
let logits = model.forward(&next_token, prompt_tokens.len())?;
Defensive patterns

Strategy: validation

Validate before calling

if pos > 0 && model_state_conv_cache_is_none() {
    anyhow::bail!("restart generation from pos 0 to seed the conv cache");
}

Try / catch

match model.forward(&tokens, pos) {
    Ok(l) => l,
    Err(e) if e.to_string().contains("empty cache despite pos > 0") => {
        // restart: seed the cache from position 0
        let _ = model.forward(&full_prompt, 0)?;
        model.forward(&tokens, pos)?
    }
    Err(e) => return Err(e.into()),
}

Prevention

When it happens

Trigger: Calling forward with pos > 0 (continuing a sequence) on a model whose conv1d_state was never initialized, was consumed/reset, or whose earlier forward ran a different branch that never populated the cache.

Common situations: Feeding prompts position-by-position without first running a forward that seeds the cache; calling forward on a reused model after a manual cache reset; mixing caching and non-caching forward paths across steps.

Related errors

AI-assisted analysis of huggingface/candle@d5fee525bf (2026-09-02). Data as JSON: /api/errors/5d4683a9bd3f18d3. Report an issue: GitHub.