risingwavelabs/risingwave · error · JoinEntryError

removing a join state entry but it is not in the cache

Error message

removing a join state entry but it is not in the cache

What it means

JoinEntryError::Remove is returned by JoinEntryState::remove when the primary key being deleted is not present in the join's in-memory cache while strict consistency is enabled. The join executor should only remove entries it previously inserted; deleting a missing key means delete/update bookkeeping has drifted from the cache contents. Without strict consistency the miss is only logged as a consistency_error and removal succeeds vacuously.

Source

Thrown at src/stream/src/executor/join/hash_join.rs:754

pub struct JoinEntryState<E: JoinEncoding> {
    /// The full copy of the state.
    cached: JoinRowSet<PkType, E::EncodedRow>,
    kv_heap_size: KvSize,
}

impl<E: JoinEncoding> EstimateSize for JoinEntryState<E> {
    fn estimated_heap_size(&self) -> usize {
        // TODO: Add btreemap internal size.
        // https://github.com/risingwavelabs/risingwave/issues/9713
        self.kv_heap_size.size()
    }
}

#[derive(Error, Debug)]
pub enum JoinEntryError {
    #[error("double inserting a join state entry")]
    Occupied,
    #[error("removing a join state entry but it is not in the cache")]
    Remove,
}

impl<E: JoinEncoding> JoinEntryState<E> {
    /// Insert into the cache.
    pub fn insert(
        &mut self,
        key: PkType,
        value: E::EncodedRow,
    ) -> Result<&mut E::EncodedRow, JoinEntryError> {
        let mut removed = false;
        if !enable_strict_consistency() {
            // strict consistency is off, let's remove existing (if any) first
            if let Some(old_value) = self.cached.remove(&key) {
                self.kv_heap_size.sub(&key, &old_value);
                removed = true;
            }
        }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check whether strict consistency is enabled for testing; disabling it downgrades this to a logged consistency_error.
2. Audit the join chunk application path for double-applied UpdateDelete or delete-after-delete sequences and fix the bookkeeping.
3. Perform a full recovery/rebuild of the join state so the cache is repopulated from the state table before processing resumes.
4. Report with the offending PK and input chunk — repeated occurrences indicate a hash-join executor bug.

Example fix

// before (errors on missing key in strict mode)
if self.cached.remove(&pk).is_none() { return Err(JoinEntryError::Remove); }
// after (tolerate missing key, log consistency error)
if let Some(value) = self.cached.remove(&pk) {
    self.kv_heap_size.sub(&pk, &value);
} else {
    consistency_error!(?pk, "removing a join state entry but it's not in the cache");
}

// before removing, ensure the key exists in the cache
if join_entry_state.get(&pk, &data_types).is_none() {
    debug_assert!(!enable_strict_consistency(), "removing a join state entry but it is not in the cache");
}

fn entry_exists<E: JoinEncoding>(state: &JoinEntryState<E>, pk: &PkType, types: &[DataType]) -> bool {
    state.get(pk, types).is_some()
}

match join_entry_state.remove(pk) {
    Err(JoinEntryError::Remove) => {
        // tolerate a duplicate delete under lenient mode; log for diagnosis
        tracing::warn!(?pk, "join state remove missed cache entry; ignoring");
    }
    other => other?,
}

When it happens

Trigger: Calling JoinEntryState::remove(pk) (hash_join executor) for a key absent from `cached` while enable_strict_consistency() is true — e.g. a duplicated UpdateDelete row, a delete applied twice, or a cache that was evicted/rebuilt while the state table still contains the key.

Common situations: Replayed delete rows after recovery; hash-join chunk application bugs where insert was skipped (e.g. duplicate update collapsed) but the corresponding delete still arrives; divergence between state-table contents and the in-memory cache under strict consistency validation.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• double inserting a join state entry
• next offset {:?} should be later than current offset {:?}
• new item epoch {} does not match current chunk offset epoch
• new item epoch {} does not exceed barrier offset epoch {}
• Division by zero

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/daabceedce736dcf. Report an issue: GitHub.