{"record":{"id":"30ef04567ce7f266","repo":"risingwavelabs/risingwave","slug":"snapshot-backfill-paused-but-received-snapshot-ch","errorCode":null,"errorMessage":"snapshot backfill paused, but received snapshot chunk","messagePattern":"snapshot backfill paused, but received snapshot chunk","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/stream/src/executor/backfill/snapshot_backfill/executor.rs","lineNumber":1126,"sourceCode":"                    {\n                        Some(config.rate_limit)\n                    } else {\n                        None\n                    }\n                });\n                yield Message::Barrier(barrier);\n                post_commit.post_yield_barrier(None).await?;\n\n                if let Some(new_rate_limit) = new_rate_limit {\n                    let new_rate_limit = new_rate_limit.into();\n                    rate_limiter.update(new_rate_limit);\n                    snapshot_stream.update_rate_limiter(new_rate_limit, chunk_size);\n                }\n            }\n            Either::Right(Some(chunk)) => {\n                if backfill_paused {\n                    return Err(\n                        anyhow!(\"snapshot backfill paused, but received snapshot chunk\").into(),\n                    );\n                }\n                rate_limiter.wait(chunk.cardinality() as _).await;\n                yield Message::Chunk(chunk);\n            }\n            Either::Right(None) => {\n                break;\n            }\n        }\n    }\n\n    // finish consuming upstream snapshot, report finish\n    let barrier_to_report_finish = receive_next_barrier(barrier_rx).await?;\n    assert_eq!(barrier_to_report_finish.epoch.prev, barrier_epoch.curr);\n    barrier_epoch = barrier_to_report_finish.epoch;\n    snapshot_stream\n        .for_vnode_pk_progress(|vnode, row_count, pk_progress| {\n            assert_eq!(pk_progress, None);","sourceCodeStart":1108,"sourceCodeEnd":1144,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/stream/src/executor/backfill/snapshot_backfill/executor.rs#L1108-L1144","documentation":"The snapshot consumption stream was paused (`backfill_paused = true`), but a snapshot chunk still arrived from the snapshot stream. This indicates a bug in pause coordination: after a pause barrier, no further snapshot chunks should be produced.","triggerScenarios":"A barrier triggers `should_start_fragment_backfill=false` (pause), yet the snapshot read stream subsequently yields a chunk in `Either::Right(Some(chunk))`.","commonSituations":"Race between the pause barrier and in-flight snapshot batches buffered in the stream; bugs in snapshot stream pause/cancel logic after operator commands like `ALTER ... PAUSE` or rate-limit updates.","solutions":["Check which barrier paused the backfill and whether the snapshot stream producer was correctly notified/cancelled at pause time.","Inspect buffered snapshot batches dispatched before the pause and ensure they are drained or cancelled before applying the pause.","If reproducible, file an issue with the fragment plan and barrier sequence — this is a pause-protocol violation."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"Either::Right(Some(chunk)) if backfill_paused => { error!(\"pause violated\"); return Err(anyhow!(\"snapshot backfill paused, but received snapshot chunk\").into()); }","preventionTips":["Ensure pause barriers cancel in-flight snapshot reads before applying the pause","Test pause/resume flows under load with buffered snapshot batches","Keep pause commands and rate-limit updates serialized with barrier processing"],"tags":["streaming","backfill","pause","invariant"],"backgroundTag":"invalid-state-transition","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}