{"record":{"id":"2ea843d9b59fec0b","repo":"risingwavelabs/risingwave","slug":"should-not-receive-barrier-with-epoch-barrier-epo","errorCode":null,"errorMessage":"should not receive barrier with epoch {barrier_epoch:?} later than snapshot epoch {snapshot_epoch}","messagePattern":"should not receive barrier with epoch (.+?) later than snapshot epoch (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/stream/src/executor/backfill/snapshot_backfill/executor.rs","lineNumber":1076,"sourceCode":"    }\n\n    let mut backfill_paused = initial_backfill_paused;\n    loop {\n        let throttle_snapshot_stream = matches!(rate_limiter.rate_limit(), RateLimit::Pause);\n        match select_barrier_and_snapshot_stream(\n            barrier_rx,\n            &mut snapshot_stream,\n            throttle_snapshot_stream,\n            backfill_paused,\n        )\n        .await?\n        {\n            Either::Left(barrier) => {\n                assert_eq!(barrier.epoch.prev, barrier_epoch.curr);\n                barrier_epoch = barrier.epoch;\n\n                if barrier_epoch.curr >= snapshot_epoch {\n                    return Err(anyhow!(\"should not receive barrier with epoch {barrier_epoch:?} later than snapshot epoch {snapshot_epoch}\").into());\n                }\n                if barrier.should_start_fragment_backfill(actor_ctx.fragment_id) {\n                    backfill_paused = false;\n                }\n                if let Some(chunk) = snapshot_stream.consume_builder() {\n                    rate_limiter.wait(chunk.cardinality() as _).await;\n                    yield Message::Chunk(chunk);\n                }\n                snapshot_stream\n                    .for_vnode_pk_progress(|vnode, row_count, pk_progress| {\n                        if let Some(pk) = pk_progress {\n                            backfill_state.update_epoch_progress(\n                                vnode,\n                                snapshot_epoch,\n                                row_count,\n                                pk,\n                            );\n                        } else {","sourceCodeStart":1058,"sourceCodeEnd":1094,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/stream/src/executor/backfill/snapshot_backfill/executor.rs#L1058-L1094","documentation":"While consuming the upstream channel of a snapshot backfill, a barrier arrived whose epoch has caught up to or passed the epoch captured for the snapshot read. This means the snapshot could no longer correspond to the assumed consistent point, so the executor aborts rather than emit inconsistent data.","triggerScenarios":"A barrier with `epoch.curr >= snapshot_epoch` is delivered to `make_consume_snapshot_stream`, e.g. snapshot reads are stalled (rate limited or paused) long enough that epochs advance past the snapshot epoch before backfill finishes consuming the snapshot.","commonSituations":"Very aggressive sync-log-store/snapshot rate limiting; long-paused backfill after config change; clock/epoch configuration anomalies causing barriers to advance while snapshot stream is stuck.","solutions":["Increase snapshot consumption throughput (raise rate limiter settings, e.g. `sync_log_store_pause_duration_ms` lower / buffer size higher).","Check whether backfill was paused (`backfill_paused`) for an unusually long time and why (pause commands, upstream slow barrier production).","If reproducible with normal settings, file an issue — a healthy pipeline should finish snapshot consumption before epochs overtake it."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// before resuming/limiting a snapshot backfill, assert epochs are still consistent\nif barrier_epoch.curr >= snapshot_epoch { return Err(anyhow!(\"snapshot epoch overtaken\").into()); }","typeGuard":null,"tryCatchPattern":"if let Err(e) = executor.execute_inner().await { log_epoch_state(); return Err(e); }","preventionTips":["Do not leave snapshot backfill paused for extended periods","Tune rate limiters so the snapshot finishes well within the snapshot epoch window","Watch for stalls in barrier consumption that let epochs advance"],"tags":["streaming","backfill","epoch","invariant"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}