should not receive barrier with epoch {barrier_epoch:?} late

Error message

should not receive barrier with epoch {barrier_epoch:?} later than snapshot epoch {snapshot_epoch}

What it means

While consuming the upstream channel of a snapshot backfill, a barrier arrived whose epoch has caught up to or passed the epoch captured for the snapshot read. This means the snapshot could no longer correspond to the assumed consistent point, so the executor aborts rather than emit inconsistent data.

Source

Thrown at src/stream/src/executor/backfill/snapshot_backfill/executor.rs:1076

    }

    let mut backfill_paused = initial_backfill_paused;
    loop {
        let throttle_snapshot_stream = matches!(rate_limiter.rate_limit(), RateLimit::Pause);
        match select_barrier_and_snapshot_stream(
            barrier_rx,
            &mut snapshot_stream,
            throttle_snapshot_stream,
            backfill_paused,
        )
        .await?
        {
            Either::Left(barrier) => {
                assert_eq!(barrier.epoch.prev, barrier_epoch.curr);
                barrier_epoch = barrier.epoch;

                if barrier_epoch.curr >= snapshot_epoch {
                    return Err(anyhow!("should not receive barrier with epoch {barrier_epoch:?} later than snapshot epoch {snapshot_epoch}").into());
                }
                if barrier.should_start_fragment_backfill(actor_ctx.fragment_id) {
                    backfill_paused = false;
                }
                if let Some(chunk) = snapshot_stream.consume_builder() {
                    rate_limiter.wait(chunk.cardinality() as _).await;
                    yield Message::Chunk(chunk);
                }
                snapshot_stream
                    .for_vnode_pk_progress(|vnode, row_count, pk_progress| {
                        if let Some(pk) = pk_progress {
                            backfill_state.update_epoch_progress(
                                vnode,
                                snapshot_epoch,
                                row_count,
                                pk,
                            );
                        } else {

View on GitHub (pinned to 6469eb736d)

Solutions

Increase snapshot consumption throughput (raise rate limiter settings, e.g. `sync_log_store_pause_duration_ms` lower / buffer size higher).
Check whether backfill was paused (`backfill_paused`) for an unusually long time and why (pause commands, upstream slow barrier production).
If reproducible with normal settings, file an issue — a healthy pipeline should finish snapshot consumption before epochs overtake it.

Defensive patterns

Strategy: validation

Validate before calling

// before resuming/limiting a snapshot backfill, assert epochs are still consistent
if barrier_epoch.curr >= snapshot_epoch { return Err(anyhow!("snapshot epoch overtaken").into()); }

Try / catch

if let Err(e) = executor.execute_inner().await { log_epoch_state(); return Err(e); }

Prevention

Do not leave snapshot backfill paused for extended periods
Tune rate limiters so the snapshot finishes well within the snapshot epoch window
Watch for stalls in barrier consumption that let epochs advance

When it happens

Trigger: A barrier with `epoch.curr >= snapshot_epoch` is delivered to `make_consume_snapshot_stream`, e.g. snapshot reads are stalled (rate limited or paused) long enough that epochs advance past the snapshot epoch before backfill finishes consuming the snapshot.

Common situations: Very aggressive sync-log-store/snapshot rate limiting; long-paused backfill after config change; clock/epoch configuration anomalies causing barriers to advance while snapshot stream is stuck.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/2ea843d9b59fec0b. Report an issue: GitHub.