risingwavelabs/risingwave · error · HummockError

Change log retention miss: table {table_id}, epoch {epoch}

Error message

Change log retention miss: table {table_id}, epoch {epoch}

What it means

Hummock (RisingWave's storage engine) reports that a change log needed to read the table's data changes at a given epoch is no longer retained. The change log (tracked in `hummock_change_log` tables) only keeps a bounded window of history; once the readable watermark epoch advances past the requested epoch, reads that require per-row change logs fail. The constructor `HummockError::change_log_retention_miss(table_id, epoch)` wraps this variant, and stream executors deliberately match it via `is_retention_or_snapshot_expired_error` to treat it as a recoverable, expected condition (e.g. to restart backfill/snapshot consumption), not a fatal fault.

Source

Thrown at src/storage/src/hummock/error.rs:51

    #[error("Encode error: {0}")]
    EncodeError(String),
    #[error("Decode error: {0}")]
    DecodeError(String),
    #[error("ObjectStore failed with IO error: {0}")]
    ObjectIoError(
        #[from]
        #[backtrace]
        ObjectError,
    ),
    #[error("Meta error: {0}")]
    MetaError(String),
    #[error("SharedBuffer error: {0}")]
    SharedBufferError(String),
    #[error("Wait epoch error: {0}")]
    WaitEpoch(String),
    #[error("Next epoch error: {0}")]
    NextEpoch(String),
    #[error("Change log retention miss: table {table_id}, epoch {epoch}")]
    ChangeLogRetentionMiss { table_id: TableId, epoch: u64 },
    #[error("Time-travel version expired: table {table_id}, epoch {epoch}")]
    TimeTravelVersionExpired { table_id: TableId, epoch: u64 },
    #[error(
        "Committed epoch mismatch: table {table_id}, committed_epoch {committed_epoch}, read_epoch {read_epoch}"
    )]
    CommittedEpochMismatch {
        table_id: TableId,
        committed_epoch: u64,
        read_epoch: u64,
    },
    #[error("Barrier read is unavailable for now. Likely the cluster is recovering")]
    ReadCurrentEpoch,
    #[error("CompactionExecutor error: {0}")]
    CompactionExecutor(String),
    #[error("FileCache error: {0}")]
    FileCache(String),
    #[error("SstObjectIdTracker error: {0}")]

View on GitHub (pinned to 6469eb736d)

Solutions

1. Restart the affected backfill/consume-upstream executor so it takes a fresh snapshot at the current epoch instead of the expired one.
2. Match on `HummockErrorInner::ChangeLogRetentionMiss` (or use `is_retention_or_snapshot_expired_error` in src/stream/src/executor/backfill/snapshot_backfill/consume_upstream/stream.rs) and handle it as an expected expiry rather than a hard failure.
3. If this occurs repeatedly, reduce the lag of change-log consumers (scale the stream graph, speed up backfill) or increase change log retention / adjust GC so the watermark advances more slowly.
4. Check cluster recovery/failover events that paused the consumer long enough for retention to expire.

Example fix

// before: propagating any hummock error as fatal
return Err(err.into());
// after: treat retention miss as expected expiry
if matches!(err.inner(), HummockErrorInner::ChangeLogRetentionMiss { .. }) {
    // restart snapshot consumption from the current readable epoch
    self.restart_from_snapshot();
    return Ok(None);
}

// Cannot be pre-validated externally: retention is enforced by the engine.
// Guard by keeping consumer lag well below the change-log retention window:
// assert!(consumer_max_lag_epochs < change_log_retention_epochs)

match err.inner() {
    HummockErrorInner::ChangeLogRetentionMiss { table_id, epoch } => {
        // expected expiry: restart snapshot/backfill consumption from the current epoch
    }
    _ => return Err(err),
}

When it happens

Trigger: A consumer of change logs (e.g. snapshot-backfill `consume_upstream` executor, or a read requesting `read_change_log` at a specific epoch) asks for change log data of `table_id` at `epoch` after the change log retention watermark for that table has moved beyond that epoch.

Common situations: Upstream MV or table with frequent versioned writes flushes and trims change logs while a downstream backfill/consume-upstream executor is stalled, paused, or restarted; a long GC interval misconfiguration lets the retention watermark advance past a slow consumer; a manual replay/rewind to an old epoch beyond retention.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Time-travel version expired: table {table_id}, epoch {epoch}
• Wait epoch error: {0}
• Next epoch error: {0}
• Committed epoch mismatch: table {table_id}, committed_epoch
• Storage error: {0}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/5f56474e8dacb1d1. Report an issue: GitHub.