{"record":{"id":"5f56474e8dacb1d1","repo":"risingwavelabs/risingwave","slug":"change-log-retention-miss-table-table-id-epoch","errorCode":null,"errorMessage":"Change log retention miss: table {table_id}, epoch {epoch}","messagePattern":"Change log retention miss: table (.+?), epoch (.+?)","errorType":"exception","errorClass":"HummockError","httpStatus":null,"severity":"error","filePath":"src/storage/src/hummock/error.rs","lineNumber":51,"sourceCode":"    #[error(\"Encode error: {0}\")]\n    EncodeError(String),\n    #[error(\"Decode error: {0}\")]\n    DecodeError(String),\n    #[error(\"ObjectStore failed with IO error: {0}\")]\n    ObjectIoError(\n        #[from]\n        #[backtrace]\n        ObjectError,\n    ),\n    #[error(\"Meta error: {0}\")]\n    MetaError(String),\n    #[error(\"SharedBuffer error: {0}\")]\n    SharedBufferError(String),\n    #[error(\"Wait epoch error: {0}\")]\n    WaitEpoch(String),\n    #[error(\"Next epoch error: {0}\")]\n    NextEpoch(String),\n    #[error(\"Change log retention miss: table {table_id}, epoch {epoch}\")]\n    ChangeLogRetentionMiss { table_id: TableId, epoch: u64 },\n    #[error(\"Time-travel version expired: table {table_id}, epoch {epoch}\")]\n    TimeTravelVersionExpired { table_id: TableId, epoch: u64 },\n    #[error(\n        \"Committed epoch mismatch: table {table_id}, committed_epoch {committed_epoch}, read_epoch {read_epoch}\"\n    )]\n    CommittedEpochMismatch {\n        table_id: TableId,\n        committed_epoch: u64,\n        read_epoch: u64,\n    },\n    #[error(\"Barrier read is unavailable for now. Likely the cluster is recovering\")]\n    ReadCurrentEpoch,\n    #[error(\"CompactionExecutor error: {0}\")]\n    CompactionExecutor(String),\n    #[error(\"FileCache error: {0}\")]\n    FileCache(String),\n    #[error(\"SstObjectIdTracker error: {0}\")]","sourceCodeStart":33,"sourceCodeEnd":69,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/storage/src/hummock/error.rs#L33-L69","documentation":"Hummock (RisingWave's storage engine) reports that a change log needed to read the table's data changes at a given epoch is no longer retained. The change log (tracked in `hummock_change_log` tables) only keeps a bounded window of history; once the readable watermark epoch advances past the requested epoch, reads that require per-row change logs fail. The constructor `HummockError::change_log_retention_miss(table_id, epoch)` wraps this variant, and stream executors deliberately match it via `is_retention_or_snapshot_expired_error` to treat it as a recoverable, expected condition (e.g. to restart backfill/snapshot consumption), not a fatal fault.","triggerScenarios":"A consumer of change logs (e.g. snapshot-backfill `consume_upstream` executor, or a read requesting `read_change_log` at a specific epoch) asks for change log data of `table_id` at `epoch` after the change log retention watermark for that table has moved beyond that epoch.","commonSituations":"Upstream MV or table with frequent versioned writes flushes and trims change logs while a downstream backfill/consume-upstream executor is stalled, paused, or restarted; a long GC interval misconfiguration lets the retention watermark advance past a slow consumer; a manual replay/rewind to an old epoch beyond retention.","solutions":["Restart the affected backfill/consume-upstream executor so it takes a fresh snapshot at the current epoch instead of the expired one.","Match on `HummockErrorInner::ChangeLogRetentionMiss` (or use `is_retention_or_snapshot_expired_error` in src/stream/src/executor/backfill/snapshot_backfill/consume_upstream/stream.rs) and handle it as an expected expiry rather than a hard failure.","If this occurs repeatedly, reduce the lag of change-log consumers (scale the stream graph, speed up backfill) or increase change log retention / adjust GC so the watermark advances more slowly.","Check cluster recovery/failover events that paused the consumer long enough for retention to expire."],"exampleFix":"// before: propagating any hummock error as fatal\nreturn Err(err.into());\n// after: treat retention miss as expected expiry\nif matches!(err.inner(), HummockErrorInner::ChangeLogRetentionMiss { .. }) {\n    // restart snapshot consumption from the current readable epoch\n    self.restart_from_snapshot();\n    return Ok(None);\n}","handlingStrategy":"try-catch","validationCode":"// Cannot be pre-validated externally: retention is enforced by the engine.\n// Guard by keeping consumer lag well below the change-log retention window:\n// assert!(consumer_max_lag_epochs < change_log_retention_epochs)","typeGuard":null,"tryCatchPattern":"match err.inner() {\n    HummockErrorInner::ChangeLogRetentionMiss { table_id, epoch } => {\n        // expected expiry: restart snapshot/backfill consumption from the current epoch\n    }\n    _ => return Err(err),\n}","preventionTips":["Keep change-log consumers (backfill, downstream MVs) running without long stalls so retention never lapses them.","Use `is_retention_or_snapshot_expired_error` instead of string matching to classify this error.","Set change log retention comfortably larger than worst-case consumer lag or restart time.","Alert on consumer watermark vs retention watermark gap."],"tags":["storage","hummock","change-log","retention","epoch"],"backgroundTag":"value-out-of-range","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}