risingwavelabs/risingwave · error · anyhow::Error

invalid commit epoch {}, prev_epoch {}

Error message

invalid commit epoch {}, prev_epoch {}

What it means

The coordinator tracks the previous committed epoch from `CommitRequest`s and rejects a commit whose epoch is lower than the previously seen one. Since commits must be monotonically non-decreasing in epoch, receiving an older epoch indicates out-of-order or duplicated commit requests. This is a strict protocol invariant of the two-phase sink commit protocol.

Source

Thrown at src/meta/src/manager/sink_coordination/handle.rs:130

    pub(super) fn poll_next_request(
        &mut self,
        cx: &mut Context<'_>,
    ) -> Poll<anyhow::Result<coordinate_request::Msg>> {
        let result = try {
            let request = ready!(self.request_stream.try_poll_next_unpin(cx))
                .ok_or_else(|| anyhow!("end of request stream"))?
                .map_err(anyhow::Error::from)?;
            let request = request.msg.ok_or_else(|| anyhow!("None msg in request"))?;
            match &request {
                coordinate_request::Msg::StartRequest(_)
                | coordinate_request::Msg::Stop(_)
                | coordinate_request::Msg::AlignInitialEpochRequest(_) => {}
                coordinate_request::Msg::CommitRequest(request) => {
                    if let Some(prev_epoch) = self.prev_epoch
                        && request.epoch < prev_epoch
                    {
                        return Poll::Ready(Err(anyhow!(
                            "invalid commit epoch {}, prev_epoch {}",
                            request.epoch,
                            prev_epoch
                        )));
                    }
                    if request.metadata.is_none() {
                        return Poll::Ready(Err(anyhow!("empty commit metadata")));
                    };
                    self.prev_epoch = Some(request.epoch);
                }
                coordinate_request::Msg::UpdateVnodeRequest(request) => {
                    let bitmap = Bitmap::from(
                        request
                            .vnode_bitmap
                            .as_ref()
                            .ok_or_else(|| anyhow!("empty vnode bitmap"))?,
                    );
                    self.vnode_bitmap = bitmap;

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the sink executor sends commits in strictly increasing epoch order and does not replay stale commits after reconnect.
2. Ensure `AlignInitialEpochRequest` is sent when a new coordinator session starts so `prev_epoch` is seeded correctly.
3. Check Hummock/epoch manager for epoch regression in the executor; fix the source of the stale epoch.
4. If triggered by duplicated retries, make commit requests idempotent (skip re-committing an already committed epoch) on the executor side.

Example fix

// executor side: skip stale commits
if epoch <= last_committed_epoch { return Ok(()); } // instead of sending CommitRequest{epoch}

// Executor side: enforce monotonic epochs before committing
if epoch <= last_committed_epoch {
    tracing::debug!(epoch, "skipping stale commit");
    return Ok(());
}

if let Err(e) = coordinator_next_request() {
    if e.to_string().contains("invalid commit epoch") {
        tracing::error!("epoch regression detected; realign initial epoch and reconnect");
    }
}

When it happens

Trigger: A `CommitRequest` arrives whose `epoch` is strictly less than `self.prev_epoch` recorded from an earlier commit on the same coordination stream.

Common situations: Sink executor retry logic replays an old commit after a partial failure; epoch alignment (`AlignInitialEpochRequest`) done incorrectly; clock/state skew after failover where a stale coordinator state persists.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• job {} in database {} has tables with different table ids. {
• empty commit metadata
• Invalid datetime: {value} {unit} is out of range
• initial epoch {} greater than aligned initial epoch {}
• newly start epoch {} after update vnode bitmap not matched w

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/8dc027bd40f639dc. Report an issue: GitHub.