risingwavelabs/risingwave · error · MetaError

since_timestamp epoch has not been resolved for snapshot bac

Error message

since_timestamp epoch has not been resolved for snapshot backfill

What it means

During a snapshot-backfill streaming job creation, the caller supplied a since_timestamp epoch whose resolution is still pending (ResolvedEpoch::resolved is None). The barrier command applier requires the concrete snapshot epoch before it can apply the CreateStreamingJob command, so it refuses to proceed. This is an ordering/state invariant: the epoch must be resolved by barrier scheduling before the command is applied.

Source

Thrown at src/meta/src/barrier/checkpoint/state.rs:547

                    &self.database_info,
                    &info.definition,
                    &info.stream_job_fragments.inner.ctx,
                    &info.streaming_job_model,
                    partial_graph_manager
                        .control_stream_manager()
                        .env
                        .actor_id_generator(),
                    worker_nodes,
                    &ensembles,
                    &info.database_resource_group,
                )?;
                {
                    assert!(!self.state.is_paused());
                    let (snapshot_epoch, since_timestamp_upstream_log_epochs) =
                        if let Some(since_epoch) = &since_epoch {
                            let (snapshot_epoch, log_epochs) =
                                since_epoch.resolved.as_ref().ok_or_else(|| {
                            MetaError::from(anyhow::anyhow!(
                                "since_timestamp epoch has not been resolved for snapshot backfill"
                            ))
                        })?;
                            (
                                *snapshot_epoch,
                                Some((
                                    log_epochs,
                                    to_partial_graph_id(self.database_id, None),
                                    barrier_info.prev_epoch(),
                                )),
                            )
                        } else {
                            (barrier_info.prev_epoch(), None)
                        };
                    // set snapshot epoch of upstream table for snapshot backfill
                    for snapshot_backfill_epoch in snapshot_backfill_info
                        .upstream_mv_table_id_to_backfill_epoch
                        .values_mut()

View on GitHub (pinned to 6469eb736d)

Solutions

1. Ensure the since_timestamp epoch is resolved (resolve_since_timestamp_log_store_epoch) before the command reaches apply_command
2. Retry creating the snapshot backfill job after the next barrier completes
3. Check barrier scheduler ordering/feature flag for since_timestamp resolution
4. Report as a bug if it reproduces consistently — resolution should precede application

Example fix

// before: apply immediately with unresolved since epoch
command.apply()
// after: wait for resolution
if since_epoch.resolved.is_none() { defer_to_next_barrier(command); }

if since_epoch.resolved.is_none() {
    return Err("defer command until since_timestamp is resolved");
}
command.apply();

fn is_resolved(e: &SinceEpoch) -> bool { e.resolved.is_some() }

match apply_command(&mut state, cmd) {
    Err(e) if e.to_string().contains("not been resolved") => defer_to_next_barrier(cmd),
    other => other,
}

When it happens

Trigger: Applying a CreateStreamingJob command with CreateStreamingJobType::SnapshotBackfill (or batch refresh with since epoch) via handle_new_barrier while since_epoch.resolved is still None — i.e. the resolution step did not run or ran after apply.

Common situations: Race between barrier resolution and command application; snapshot backfill created on a database whose barrier scheduler has not yet resolved the since_timestamp; internal scheduling bugs after upgrades touching barrier state.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• since_timestamp requires at least one upstream table
• The cluster is bootstrapping
• The cluster is recovering
• The cluster is recovering-adhoc
• should get metadata on checkpoint barrier

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/8bf86afe92619c88. Report an issue: GitHub.