{"record":{"id":"59c5058b5efe7bef","repo":"risingwavelabs/risingwave","slug":"truncation-epoch-should-not-be-larger-than-curr","errorCode":null,"errorMessage":"truncation epoch {} should not be larger than current epoch {}","messagePattern":"truncation epoch (.+?) should not be larger than current epoch (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/stream/src/common/log_store_impl/kv_log_store/buffer.rs","lineNumber":362,"sourceCode":"        {\n            ret = inner.truncation_list.pop_front();\n        }\n        ret\n    }\n\n    pub(crate) async fn wait_for_barrier_truncation(\n        &self,\n        curr_epoch: u64,\n    ) -> LogStoreResult<ReaderTruncationOffsetType> {\n        loop {\n            let notified = self.truncate_notify.notified();\n\n            {\n                let mut inner = self.buffer.inner();\n                while let Some((epoch, seq_id)) = inner.truncation_list.pop_front() {\n                    if epoch > curr_epoch {\n                        // TODO: should panic, after we confirm the correctness\n                        return Err(anyhow::anyhow!(\n                            \"truncation epoch {} should not be larger than current epoch {}\",\n                            epoch,\n                            curr_epoch\n                        ));\n                    }\n                    if epoch == curr_epoch && seq_id.is_none() {\n                        return Ok((epoch, seq_id));\n                    }\n                }\n            }\n\n            notified\n                .instrument_await(\"Wait For Barrier Truncation\")\n                .await;\n        }\n    }\n\n    pub(crate) fn flush_all_unflushed(","sourceCodeStart":344,"sourceCodeEnd":380,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/stream/src/common/log_store_impl/kv_log_store/buffer.rs#L344-L380","documentation":"wait_for_barrier_truncation drains the buffer's truncation_list, applying pending truncate requests. This error is returned when a queued truncation epoch is greater than the writer's current epoch, which should be impossible: you cannot truncate data that has not been written yet. It is a safety check (marked TODO panic) guarding the epoch/sequence bookkeeping in the kv log store buffer.","triggerScenarios":"A truncation request (usually from barrier handling or reader truncate) enqueued with an epoch larger than `curr_epoch` at the time the buffer processes the truncation list.","commonSituations":"Epoch ordering bugs after failover or recovery (stale/newer barrier mixed into the truncation list); races between truncation from a new reader instance and an old writer epoch; custom code or tests manipulating truncation offsets out of order.","solutions":["Log the full truncation_list and curr_epoch to identify which epoch is out of order.","Check for concurrent/overlapping reader or writer instances submitting truncations after failover.","Verify barrier epoch ordering logic — a recovered barrier with a higher epoch must not truncate an older writer's buffer.","Ensure the writer's curr_epoch is updated before truncations for that epoch are enqueued."],"exampleFix":"// before: truncating before the epoch is current\nwriter.enqueue_truncation(new_epoch)?; // new_epoch > curr_epoch\n\n// after: advance the epoch first\nwriter.flush_current_epoch(new_epoch, opts).await?;\nwriter.enqueue_truncation(new_epoch)?;","handlingStrategy":"validation","validationCode":"// before enqueueing a truncation\nif epoch > writer.curr_epoch() {\n    return Err(\"cannot truncate an epoch that is not yet current\");\n}\nwriter.enqueue_truncation(epoch, seq_id)?;","typeGuard":"fn truncation_in_order(epoch: u64, curr_epoch: u64) -> bool { epoch <= curr_epoch }","tryCatchPattern":"match buffer.wait_for_barrier_truncation(curr_epoch).await {\n    Err(e) if e.to_string().contains(\"should not be larger than current epoch\") => {\n        log_epoch_ordering_violation(e); // investigate failover/epoch bookkeeping\n    }\n    r => r?,\n}","preventionTips":["Advance curr_epoch (flush/barrier) before enqueueing truncations for that epoch.","Guard against duplicate writers/readers after failover.","Add assertions in tests that truncation epochs are monotonically ordered."],"tags":["epoch","truncation","invariant","log-store"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}