risingwavelabs/risingwave · error

snapshot backfill epoch set again: {} {} {}

Error message

snapshot backfill epoch set again: {} {} {}

What it means

During stream graph building, the meta service records the snapshot epoch used for a table's snapshot backfill stream scan. `fill_snapshot_backfill_epoch` asserts that each upstream table's stream scan gets exactly one snapshot epoch; if `replace()` finds a previously set epoch for the same table, the graph builder throws. This is an internal invariant: a fragment should never scan the same upstream table twice with different (or duplicate) snapshot backfill epochs.

Source

Thrown at src/meta/src/stream/stream_graph/fragment.rs:1627

        {
            result = try {
                let table_id = stream_scan.table_id;
                let snapshot_epoch = cross_db_snapshot_backfill_info
                    .upstream_mv_table_id_to_backfill_epoch
                    .get(&table_id)
                    .or_else(|| {
                        snapshot_backfill_info.and_then(|snapshot_backfill_info| {
                            snapshot_backfill_info
                                .upstream_mv_table_id_to_backfill_epoch
                                .get(&table_id)
                        })
                    })
                    .ok_or_else(|| anyhow!("upstream table id not covered: {}", table_id))?
                    .ok_or_else(|| anyhow!("upstream table id not set: {}", table_id))?;
                if let Some(prev_snapshot_epoch) =
                    stream_scan.snapshot_backfill_epoch.replace(snapshot_epoch)
                {
                    Err(anyhow!(
                        "snapshot backfill epoch set again: {} {} {}",
                        table_id,
                        prev_snapshot_epoch,
                        snapshot_epoch
                    ))?;
                }
                applied = true;
            };
            result.is_ok()
        } else {
            true
        }
    });
    result.map_err(MetaError::from).map(|_| applied)
}

static EMPTY_HASHMAP: LazyLock<HashMap<GlobalFragmentId, StreamFragmentEdge>> =
    LazyLock::new(HashMap::new);

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the built graph (fragment plan) and remove the duplicate stream scan over the same upstream table so each table id is scanned once for snapshot backfill.
2. If duplicate reference is intentional, restructure the job so downstream fragments reuse a single upstream fragment instead of directly scanning the table twice.
3. Check for recent changes to `fill_snapshot_backfill_epoch` call sites and ensure only one call happens per table id; file a bug with the SQL/job definition if triggered by an ordinary query.

Example fix

// before: two stream scans both call fill_snapshot_backfill_epoch for table_id 1001
// after: reuse one stream scan fragment's output for both consumers
// (restructure the graph so only one stream scan references table_id 1001)

// Before building the graph, ensure each upstream table id appears in exactly one stream scan
let mut seen = std::collections::HashSet::new();
for scan in collect_stream_scans(&graph) {
    if !seen.insert(scan.upstream_table_id) {
        return Err(format!("table {} scanned by multiple stream scans", scan.upstream_table_id));
    }
}

When it happens

Trigger: Building a graph where a streaming job has two stream-scan operators reading the same upstream table id (e.g. a job created via CREATE TABLE AS or a sink referencing the same MV table from multiple upstream fragments), so `fill_snapshot_backfill_epoch` is invoked twice for one `table_id` with an epoch already recorded.

Common situations: Rare meta-side invariant violations surfaced during CREATE MATERIALIZED VIEW / CREATE SINK graph construction, typically after refactors to stream-scan/upstream-table wiring or during upgrading where a job references the same upstream table from multiple fragments.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• unexpected referring object type: {}
• table_fragment does not exist: id={0}
• downstream relation missing for {} -> {}
• iceberg pk-index writer input must be a merge after actor re
• the upstream fragment should be a MView or Source, got fragm

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/65df22fd2e2f78e4. Report an issue: GitHub.