risingwavelabs/risingwave · error · anyhow::Error

recovered snapshot backfill job {} has no snapshot backfill

Error message

recovered snapshot backfill job {} has no snapshot backfill info

What it means

Same invariant as the batch-refresh variant but on the recovery path of `inject_database_initial_barrier`: a snapshot backfill job being recovered must expose snapshot backfill info derived from its persisted fragments; when the lookup returns None the recovery aborts with this error. It indicates the recovered job's fragment metadata lacks the expected snapshot backfill node.

Source

Thrown at src/meta/src/barrier/rpc.rs:825

            );
            if committed_epoch == barrier_info.prev_epoch() {
                info!(
                    "recovered creating snapshot backfill job {} catch up with upstream already",
                    job_id
                );
                database_jobs
                    .try_insert(job_id, (fragment_infos, true))
                    .expect("non-duplicate");
                continue;
            }
            let snapshot_backfill_info = StreamFragmentGraph::collect_snapshot_backfill_info_impl(
                fragment_infos
                    .values()
                    .map(|fragment| (&fragment.nodes, fragment.fragment_type_mask)),
            )?
            .0
            .ok_or_else(|| {
                anyhow!(
                    "recovered snapshot backfill job {} has no snapshot backfill info",
                    job_id
                )
            })?;
            let mut snapshot_epoch = None;
            let upstream_table_ids: HashSet<_> = snapshot_backfill_info
                .upstream_mv_table_id_to_backfill_epoch
                .keys()
                .cloned()
                .collect();
            for (upstream_table_id, epoch) in
                snapshot_backfill_info.upstream_mv_table_id_to_backfill_epoch
            {
                let epoch = epoch.ok_or_else(|| anyhow!("recovered snapshot backfill job {} to upstream {} has not set snapshot epoch", job_id, upstream_table_id))?;
                let snapshot_epoch = snapshot_epoch.get_or_insert(epoch);
                if *snapshot_epoch != epoch {
                    return Err(anyhow!("snapshot epoch {} to upstream {} different to snapshot epoch {} to previous upstream", epoch, upstream_table_id, snapshot_epoch).into());
                }

View on GitHub (pinned to 6469eb736d)

Solutions

  1. Dump the job's fragment infos from the meta store and verify a snapshot backfill fragment exists.
  2. Restore meta state from a consistent snapshot/backup if recovery state is corrupted.
  3. Upgrade consistently: ensure all persisted jobs were created by a version that persists snapshot backfill info.
  4. If unrecoverable, drop and recreate the affected job.
Defensive patterns

Strategy: validation

Validate before calling

// On recovery, pre-check persisted fragment infos before injecting the barrier
if !fragment_infos.values().any(|f| {
    f.nodes.values().any(|n| n.snapshot_backfill_info.is_some())
}) {
    // route to a repair/recreate path instead of failing injection
}

Prevention

When it happens

Trigger: Recovery/inject-initial-barrier over a snapshot backfill job whose persisted fragment infos contain no fragment with snapshot backfill type info.

Common situations: Meta recovery after crash/upgrade; metadata written by an older version without snapshot backfill info; corrupted or partially persisted fragment graph for the job.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/9ecfb06f84fb377d. Report an issue: GitHub.