risingwavelabs/risingwave · error

snapshot backfill paused, but received snapshot chunk

Error message

snapshot backfill paused, but received snapshot chunk

What it means

The snapshot consumption stream was paused (`backfill_paused = true`), but a snapshot chunk still arrived from the snapshot stream. This indicates a bug in pause coordination: after a pause barrier, no further snapshot chunks should be produced.

Source

Thrown at src/stream/src/executor/backfill/snapshot_backfill/executor.rs:1126

                    {
                        Some(config.rate_limit)
                    } else {
                        None
                    }
                });
                yield Message::Barrier(barrier);
                post_commit.post_yield_barrier(None).await?;

                if let Some(new_rate_limit) = new_rate_limit {
                    let new_rate_limit = new_rate_limit.into();
                    rate_limiter.update(new_rate_limit);
                    snapshot_stream.update_rate_limiter(new_rate_limit, chunk_size);
                }
            }
            Either::Right(Some(chunk)) => {
                if backfill_paused {
                    return Err(
                        anyhow!("snapshot backfill paused, but received snapshot chunk").into(),
                    );
                }
                rate_limiter.wait(chunk.cardinality() as _).await;
                yield Message::Chunk(chunk);
            }
            Either::Right(None) => {
                break;
            }
        }
    }

    // finish consuming upstream snapshot, report finish
    let barrier_to_report_finish = receive_next_barrier(barrier_rx).await?;
    assert_eq!(barrier_to_report_finish.epoch.prev, barrier_epoch.curr);
    barrier_epoch = barrier_to_report_finish.epoch;
    snapshot_stream
        .for_vnode_pk_progress(|vnode, row_count, pk_progress| {
            assert_eq!(pk_progress, None);

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check which barrier paused the backfill and whether the snapshot stream producer was correctly notified/cancelled at pause time.
2. Inspect buffered snapshot batches dispatched before the pause and ensure they are drained or cancelled before applying the pause.
3. If reproducible, file an issue with the fragment plan and barrier sequence — this is a pause-protocol violation.

Either::Right(Some(chunk)) if backfill_paused => { error!("pause violated"); return Err(anyhow!("snapshot backfill paused, but received snapshot chunk").into()); }

When it happens

Trigger: A barrier triggers `should_start_fragment_backfill=false` (pause), yet the snapshot read stream subsequently yields a chunk in `Either::Right(Some(chunk))`.

Common situations: Race between the pause barrier and in-flight snapshot batches buffered in the stream; bugs in snapshot stream pause/cancel logic after operator commands like `ALTER ... PAUSE` or rate-limit updates.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• should not receive barrier with epoch {barrier_epoch:?} late
• core predicate must exist
• batch refresh job {} has no snapshot backfill info
• recovered snapshot backfill job {} has no snapshot backfill
• should not pause when having snapshot backfill job {job_id}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/30ef04567ce7f266. Report an issue: GitHub.