risingwavelabs/risingwave · error · anyhow::Error

snapshot epoch {} to upstream {} different to snapshot epoch

Error message

snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream

What it means

During recovery, all upstream MVs of a snapshot backfill job must agree on a single snapshot epoch. The code keeps the first seen epoch via `get_or_insert` and throws this error when a later upstream's epoch differs from the accumulated one. Mixed epochs mean the job's upstreams were snapshotted inconsistently and recovery cannot proceed.

Source

Thrown at src/meta/src/barrier/rpc.rs:842

            .ok_or_else(|| {
                anyhow!(
                    "recovered snapshot backfill job {} has no snapshot backfill info",
                    job_id
                )
            })?;
            let mut snapshot_epoch = None;
            let upstream_table_ids: HashSet<_> = snapshot_backfill_info
                .upstream_mv_table_id_to_backfill_epoch
                .keys()
                .cloned()
                .collect();
            for (upstream_table_id, epoch) in
                snapshot_backfill_info.upstream_mv_table_id_to_backfill_epoch
            {
                let epoch = epoch.ok_or_else(|| anyhow!("recovered snapshot backfill job {} to upstream {} has not set snapshot epoch", job_id, upstream_table_id))?;
                let snapshot_epoch = snapshot_epoch.get_or_insert(epoch);
                if *snapshot_epoch != epoch {
                    return Err(anyhow!("snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream", epoch, upstream_table_id, snapshot_epoch).into());
                }
            }
            let snapshot_epoch = snapshot_epoch.ok_or_else(|| {
                anyhow!(
                    "snapshot backfill job {} has not set snapshot epoch",
                    job_id
                )
            })?;
            for upstream_table_id in &upstream_table_ids {
                subscribers
                    .entry(*upstream_table_id)
                    .or_default()
                    .try_insert(job_id.as_subscriber_id(), SubscriberType::SnapshotBackfill)
                    .expect("non-duplicate");
            }
            ongoing_snapshot_backfill_jobs
                .try_insert(
                    job_id,

View on GitHub (pinned to 6469eb736d)

Solutions

Inspect the per-upstream epochs in the job's snapshot backfill info to identify the inconsistent upstream.
Recreate the snapshot backfill job so all upstreams are captured in one barrier/epoch.
Restore a consistent meta snapshot from before the divergence.
File a bug with the epoch values and job_id if this appears on an untouched job.

Defensive patterns

Strategy: validation

Validate before calling

// Confirm all upstreams share one epoch before recovery
let epochs: HashSet<_> = snapshot_backfill_info
    .upstream_mv_table_id_to_backfill_epoch
    .values()
    .flatten()
    .copied()
    .collect();
assert!(epochs.len() <= 1, "inconsistent snapshot epochs: {epochs:?}");

Prevention

Capture all upstream MVs within a single barrier so they share one snapshot epoch.
Add an invariant check when persisting snapshot backfill info.
Log per-upstream epochs to make divergence diagnosable.

When it happens

Trigger: `inject_database_initial_barrier` recovering a snapshot backfill job whose `upstream_mv_table_id_to_backfill_epoch` contains two different epoch values across upstream tables.

Common situations: Partial/concurrent upstream registration across different barrier epochs; metadata corruption; jobs mutated by mismatched code versions.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/35e63c5391e068aa. Report an issue: GitHub.