risingwavelabs/risingwave · error · anyhow::Error

recovered snapshot backfill job

Error message

recovered snapshot backfill job {} has no snapshot backfill info

What it means

Same invariant as the batch-refresh variant but on the recovery path of `inject_database_initial_barrier`: a snapshot backfill job being recovered must expose snapshot backfill info derived from its persisted fragments; when the lookup returns None the recovery aborts with this error. It indicates the recovered job's fragment metadata lacks the expected snapshot backfill node.

Solutions

  1. Dump the job's fragment infos from the meta store and verify a snapshot backfill fragment exists.
  2. Restore meta state from a consistent snapshot/backup if recovery state is corrupted.
  3. Upgrade consistently: ensure all persisted jobs were created by a version that persists snapshot backfill info.
  4. If unrecoverable, drop and recreate the affected job.
Defensive patterns

Strategy: validation

Validate before calling

// On recovery, pre-check persisted fragment infos before injecting the barrier
if !fragment_infos.values().any(|f| {
    f.nodes.values().any(|n| n.snapshot_backfill_info.is_some())
}) {
    // route to a repair/recreate path instead of failing injection
}

Prevention

When it happens

Trigger: Recovery/inject-initial-barrier over a snapshot backfill job whose persisted fragment infos contain no fragment with snapshot backfill type info.

Common situations: Meta recovery after crash/upgrade; metadata written by an older version without snapshot backfill info; corrupted or partially persisted fragment graph for the job.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/9ecfb06f84fb377d. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/barrier/rpc.rs:825

            );
            if committed_epoch == barrier_info.prev_epoch() {
                info!(
                    "recovered creating snapshot backfill job {} catch up with upstream already",
                    job_id
                );
                database_jobs
                    .try_insert(job_id, (fragment_infos, true))
                    .expect("non-duplicate");
                continue;
            }
            let snapshot_backfill_info = StreamFragmentGraph::collect_snapshot_backfill_info_impl(
                fragment_infos
                    .values()
                    .map(|fragment| (&fragment.nodes, fragment.fragment_type_mask)),
            )?
            .0
            .ok_or_else(|| {
                anyhow!(
                    "recovered snapshot backfill job {} has no snapshot backfill info",
                    job_id
                )
            })?;
            let mut snapshot_epoch = None;
            let upstream_table_ids: HashSet<_> = snapshot_backfill_info
                .upstream_mv_table_id_to_backfill_epoch
                .keys()
                .cloned()
                .collect();
            for (upstream_table_id, epoch) in
                snapshot_backfill_info.upstream_mv_table_id_to_backfill_epoch
            {
                let epoch = epoch.ok_or_else(|| anyhow!("recovered snapshot backfill job {} to upstream {} has not set snapshot epoch", job_id, upstream_table_id))?;
                let snapshot_epoch = snapshot_epoch.get_or_insert(epoch);
                if *snapshot_epoch != epoch {
                    return Err(anyhow!("snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream", epoch, upstream_table_id, snapshot_epoch).into());
                }

View on GitHub (pinned to 6469eb736d)