risingwavelabs/risingwave · error · anyhow::Error

snapshot backfill job {} has not set snapshot epoch

Error message

snapshot backfill job {} has not set snapshot epoch

What it means

After validating all per-upstream epochs, recovery consolidates a single snapshot epoch with `snapshot_epoch.get_or_insert(epoch)` inside the loop; if the loop body never ran (the map was empty), the Option is still None and `ok_or_else` throws this error. A snapshot backfill job with no upstream epochs recorded cannot determine its snapshot point, so recovery aborts.

Source

Thrown at src/meta/src/barrier/rpc.rs:846

                )
            })?;
            let mut snapshot_epoch = None;
            let upstream_table_ids: HashSet<_> = snapshot_backfill_info
                .upstream_mv_table_id_to_backfill_epoch
                .keys()
                .cloned()
                .collect();
            for (upstream_table_id, epoch) in
                snapshot_backfill_info.upstream_mv_table_id_to_backfill_epoch
            {
                let epoch = epoch.ok_or_else(|| anyhow!("recovered snapshot backfill job {} to upstream {} has not set snapshot epoch", job_id, upstream_table_id))?;
                let snapshot_epoch = snapshot_epoch.get_or_insert(epoch);
                if *snapshot_epoch != epoch {
                    return Err(anyhow!("snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream", epoch, upstream_table_id, snapshot_epoch).into());
                }
            }
            let snapshot_epoch = snapshot_epoch.ok_or_else(|| {
                anyhow!(
                    "snapshot backfill job {} has not set snapshot epoch",
                    job_id
                )
            })?;
            for upstream_table_id in &upstream_table_ids {
                subscribers
                    .entry(*upstream_table_id)
                    .or_default()
                    .try_insert(job_id.as_subscriber_id(), SubscriberType::SnapshotBackfill)
                    .expect("non-duplicate");
            }
            ongoing_snapshot_backfill_jobs
                .try_insert(
                    job_id,
                    (
                        fragment_infos,
                        upstream_table_ids,
                        committed_epoch,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Verify the job's upstream mapping in the meta store; it must be non-empty.
2. Recreate the snapshot backfill job if its persisted state is empty.
3. Restore meta from a backup taken when the job was fully initialized.
4. Report a bug if the job had upstreams before the crash/recovery.

// Ensure the upstream map is non-empty before recovery
if snapshot_backfill_info
    .upstream_mv_table_id_to_backfill_epoch
    .is_empty()
{
    // recreate job or fail fast with a clear message
}

When it happens

Trigger: Recovering a snapshot backfill job whose `upstream_mv_table_id_to_backfill_epoch` map is empty, leaving `snapshot_epoch` unset.

Common situations: Job persisted before any upstream was registered; metadata corruption or truncation; upstream list cleared by a faulty recovery path.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• recovered snapshot backfill job {} to upstream {} has not se
• snapshot epoch {} to upstream {} different to snapshot epoch
• job {} in database {} has tables with different table ids. {
• recovered snapshot backfill job {} has no snapshot backfill
• cluster under recovery[{}]

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/b2291b497800390d. Report an issue: GitHub.