risingwavelabs/risingwave · error · anyhow::Error

recovered snapshot backfill job {} to upstream {} has not se

Error message

recovered snapshot backfill job {} to upstream {} has not set snapshot epoch

What it means

While recovering a snapshot backfill job, `inject_database_initial_barrier` iterates `upstream_mv_table_id_to_backfill_epoch`; each upstream entry's epoch is an `Option` and must be set. `epoch.ok_or_else(...)` throws this error when some upstream MV still has no snapshot epoch recorded. It means the backfill mapping is present but incomplete.

Source

Thrown at src/meta/src/barrier/rpc.rs:839

                    .map(|fragment| (&fragment.nodes, fragment.fragment_type_mask)),
            )?
            .0
            .ok_or_else(|| {
                anyhow!(
                    "recovered snapshot backfill job {} has no snapshot backfill info",
                    job_id
                )
            })?;
            let mut snapshot_epoch = None;
            let upstream_table_ids: HashSet<_> = snapshot_backfill_info
                .upstream_mv_table_id_to_backfill_epoch
                .keys()
                .cloned()
                .collect();
            for (upstream_table_id, epoch) in
                snapshot_backfill_info.upstream_mv_table_id_to_backfill_epoch
            {
                let epoch = epoch.ok_or_else(|| anyhow!("recovered snapshot backfill job {} to upstream {} has not set snapshot epoch", job_id, upstream_table_id))?;
                let snapshot_epoch = snapshot_epoch.get_or_insert(epoch);
                if *snapshot_epoch != epoch {
                    return Err(anyhow!("snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream", epoch, upstream_table_id, snapshot_epoch).into());
                }
            }
            let snapshot_epoch = snapshot_epoch.ok_or_else(|| {
                anyhow!(
                    "snapshot backfill job {} has not set snapshot epoch",
                    job_id
                )
            })?;
            for upstream_table_id in &upstream_table_ids {
                subscribers
                    .entry(*upstream_table_id)
                    .or_default()
                    .try_insert(job_id.as_subscriber_id(), SubscriberType::SnapshotBackfill)
                    .expect("non-duplicate");
            }

View on GitHub (pinned to 6469eb736d)

Solutions

Recover from an earlier consistent meta snapshot taken before the partial write.
Recreate the snapshot backfill job so all upstream epochs are assigned atomically.
Check meta logs to find which upstream table was incomplete and inspect the writer path.
Report as a bug if a completed job shows unset epochs after clean recovery.

Defensive patterns

Strategy: validation

Validate before calling

// Verify every upstream epoch is set before attempting recovery
let all_set = snapshot_backfill_info
    .upstream_mv_table_id_to_backfill_epoch
    .values()
    .all(|e| e.is_some());
if !all_set {
    // repair or recreate the job before injection
}

Type guard

// Rust: Option<Epoch> narrowing
fn set_epochs(m: &HashMap<TableId, Option<Epoch>>) -> Option<HashMap<TableId, Epoch>> {
    m.iter().map(|(k, v)| Some((*k, v?))).collect()
}

Prevention

Assign upstream epochs atomically when building snapshot backfill info.
Never persist a job whose upstream map contains None epochs.
Alert on jobs whose backfill epochs stay unset beyond initialization.

When it happens

Trigger: Recovery encountering a snapshot backfill job where at least one entry in `upstream_mv_table_id_to_backfill_epoch` maps to `None`.

Common situations: Job persisted mid-initialization (upstreams registered, epochs not yet assigned); crash between registering upstreams and setting epochs; state written by a buggy or older code path.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/9517cde88adf2879. Report an issue: GitHub.