risingwavelabs/risingwave · error

truncation epoch {} should not be larger than current epoch

Error message

truncation epoch {} should not be larger than current epoch {}

What it means

wait_for_barrier_truncation drains the buffer's truncation_list, applying pending truncate requests. This error is returned when a queued truncation epoch is greater than the writer's current epoch, which should be impossible: you cannot truncate data that has not been written yet. It is a safety check (marked TODO panic) guarding the epoch/sequence bookkeeping in the kv log store buffer.

Source

Thrown at src/stream/src/common/log_store_impl/kv_log_store/buffer.rs:362

        {
            ret = inner.truncation_list.pop_front();
        }
        ret
    }

    pub(crate) async fn wait_for_barrier_truncation(
        &self,
        curr_epoch: u64,
    ) -> LogStoreResult<ReaderTruncationOffsetType> {
        loop {
            let notified = self.truncate_notify.notified();

            {
                let mut inner = self.buffer.inner();
                while let Some((epoch, seq_id)) = inner.truncation_list.pop_front() {
                    if epoch > curr_epoch {
                        // TODO: should panic, after we confirm the correctness
                        return Err(anyhow::anyhow!(
                            "truncation epoch {} should not be larger than current epoch {}",
                            epoch,
                            curr_epoch
                        ));
                    }
                    if epoch == curr_epoch && seq_id.is_none() {
                        return Ok((epoch, seq_id));
                    }
                }
            }

            notified
                .instrument_await("Wait For Barrier Truncation")
                .await;
        }
    }

    pub(crate) fn flush_all_unflushed(

View on GitHub (pinned to 6469eb736d)

Solutions

1. Log the full truncation_list and curr_epoch to identify which epoch is out of order.
2. Check for concurrent/overlapping reader or writer instances submitting truncations after failover.
3. Verify barrier epoch ordering logic — a recovered barrier with a higher epoch must not truncate an older writer's buffer.
4. Ensure the writer's curr_epoch is updated before truncations for that epoch are enqueued.

Example fix

// before: truncating before the epoch is current
writer.enqueue_truncation(new_epoch)?; // new_epoch > curr_epoch

// after: advance the epoch first
writer.flush_current_epoch(new_epoch, opts).await?;
writer.enqueue_truncation(new_epoch)?;

// before enqueueing a truncation
if epoch > writer.curr_epoch() {
    return Err("cannot truncate an epoch that is not yet current");
}
writer.enqueue_truncation(epoch, seq_id)?;

fn truncation_in_order(epoch: u64, curr_epoch: u64) -> bool { epoch <= curr_epoch }

match buffer.wait_for_barrier_truncation(curr_epoch).await {
    Err(e) if e.to_string().contains("should not be larger than current epoch") => {
        log_epoch_ordering_violation(e); // investigate failover/epoch bookkeeping
    }
    r => r?,
}

When it happens

Trigger: A truncation request (usually from barrier handling or reader truncate) enqueued with an epoch larger than `curr_epoch` at the time the buffer processes the truncation list.

Common situations: Epoch ordering bugs after failover or recovery (stale/newer barrier mixed into the truncation list); races between truncation from a new reader instance and an old writer epoch; custom code or tests manipulating truncation offsets out of order.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• truncate at a later offset {:?} than the current latest offs
• Iceberg metadata scan should not have input executors
• Chunk size can't be zero!
• initial epoch {} greater than aligned initial epoch {}
• missing lagging barriers for direct log-store start from sna

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/59c5058b5efe7bef. Report an issue: GitHub.