risingwavelabs/risingwave · error

job has inconsistent snapshot epochs for upstream table

Error message

job {} has inconsistent snapshot epochs for upstream table {}

What it means

Within one streaming job, multiple fragments that scan the same upstream table (snapshot backfill scans) recorded different snapshot backfill epochs. The job's persisted fragments must agree on a single snapshot epoch per upstream table; disagreement is an internal invariant violation detected while computing change-log truncate info.

Solutions

1. Drop and recreate the affected streaming job listed in the error.
2. Check for known snapshot-backfill bugs in your RisingWave version and upgrade to a patched release.
3. Restore metadata store from a backup taken before the inconsistency appeared.
4. Report to maintainers with job_id and fragment ids; this indicates an internal bug.

match get_table_change_log_truncate_info().await {
    Err(e) if e.to_string().contains("inconsistent snapshot epochs") => /* recreate the job or restore metadata; do not blindly retry */,
    other => other?,
}

When it happens

Trigger: Calling get_table_change_log_truncate_info when two fragments of the same job have StreamScan nodes for the same table_id with differing snapshot_backfill_epoch values.

Common situations: Corrupted or partially-updated persisted fragment graph, buggy snapshot backfill scheduling, or version-skew during upgrades.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• cannot get database_id of fragment
• inconsistent hummock version: expected
• job in database has no state table after recovery
• job in database has tables with different table ids.
• since_timestamp epoch has not been resolved for snapshot…

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/3d8b933f60a3827a. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/controller/streaming_job.rs:327

                        Ok(scan_type) => scan_type,
                        Err(err) => {
                            collection_error = Some(anyhow::Error::new(err).context(format!(
                                "invalid persisted stream scan type {} in job {} fragment {}",
                                stream_scan.stream_scan_type, fragment.job_id, fragment.fragment_id
                            )));
                            return;
                        }
                    };
                    if scan_type != StreamScanType::SnapshotBackfill {
                        return;
                    }
                    match info
                        .upstream_table_snapshot_epochs
                        .entry(stream_scan.table_id)
                    {
                        std::collections::hash_map::Entry::Occupied(entry) => {
                            if entry.get() != &stream_scan.snapshot_backfill_epoch {
                                collection_error = Some(anyhow!(
                                    "job {} has inconsistent snapshot epochs for upstream table {}",
                                    fragment.job_id,
                                    stream_scan.table_id
                                ));
                            }
                        }
                        std::collections::hash_map::Entry::Vacant(entry) => {
                            entry.insert(stream_scan.snapshot_backfill_epoch);
                        }
                    }
                });
                if let Some(err) = collection_error {
                    return Err(err.into());
                }
            }
        }
        let independent_jobs = job_info
            .into_values()

View on GitHub (pinned to 6469eb736d)