risingwavelabs/risingwave · error

Iceberg branch {} disappeared after manifest rewrite for sin

Error message

Iceberg branch {} disappeared after manifest rewrite for sink {}

What it means

After committing an Iceberg manifest-rewrite transaction, the code looks up the branch's snapshot in the rewritten table metadata. If the branch reference is gone from the table metadata despite the commit succeeding, the internal invariant is broken (concurrent modification or catalog divergence), and GC fails with this error for the given sink.

Source

Thrown at src/meta/src/manager/iceberg_compaction/gc.rs:441

        let estimated_output_manifest_count = plan.estimated_output_manifest_count;
        let rewrite_paths = plan.rewrite_paths;
        let txn = Transaction::new(&table);
        let tx = txn
            .rewrite_manifests()
            .rewrite_if(Box::new(move |manifest| {
                rewrite_paths.contains(&manifest.manifest_path)
            }))
            .cluster_by(Box::new(|_| "risingwave-maintenance".to_owned()))
            .set_target_branch(branch.clone())
            .apply(txn)
            .map_err(|e| SinkError::Iceberg(e.into()))?;
        let table = tx
            .commit(catalog.as_ref())
            .await
            .map_err(|e| SinkError::Iceberg(e.into()))?;

        let Some(rewritten_snapshot) = table.metadata().snapshot_for_ref(&branch) else {
            return Err(anyhow!(
                "Iceberg branch {} disappeared after manifest rewrite for sink {}",
                branch,
                sink_id
            )
            .into());
        };
        if rewritten_snapshot.snapshot_id() == current_snapshot_id {
            tracing::warn!(
                iceberg_component = "manifest_maintenance",
                iceberg_operation = "rewrite_manifests",
                table = %table_ident,
                %sink_id,
                %branch,
                current_snapshot_id,
                selected_manifest_count,
                "Manifest rewrite completed without creating a new snapshot",
            );
            return Ok(());

View on GitHub (pinned to 6469eb736d)

Solutions

1. Re-run the GC/compaction cycle for the sink; the branch lookup will be re-attempted against fresh metadata.
2. Verify with the catalog that the branch still exists; recreate it if external tooling dropped it.
3. Ensure only one process performs manifest rewrite (locking/leader election) to avoid concurrent branch mutation.
4. Check other maintenance jobs' logs around the commit timestamp for branch removal.

// Before rewrite: confirm the branch exists and no other maintainer job runs concurrently
let snapshot = table.metadata().snapshot_for_ref(&branch)
    .ok_or_else(|| anyhow!("branch {} missing before rewrite", branch))?;

match check_and_rewrite_manifests(...).await {
    Err(e) if e.to_string().contains("disappeared after manifest rewrite") => {
        warn!("branch ref changed concurrently; re-running GC");
        retry_with_backoff(3, || check_and_rewrite_manifests(...)).await
    }
    other => other,
}

When it happens

Trigger: check_and_rewrite_manifests (invoked by perform_gc_operations) commits a manifest rewrite, then snapshot_for_ref(&branch) returns None because another process dropped/overwrote the branch ref between read and commit.

Common situations: Concurrent Iceberg GC/expire-snapshots jobs run by multiple meta processes; an external tool (Spark/Flink/Trino maintenance) removed the branch while RisingWave was rewriting manifests; misconfigured branch names changed outside RW.

Related errors

• manual iceberg compaction is already waiting for sink {}
• Manual iceberg compaction task {} for sink {} failed: {}
• No iceberg compactor available
• Manual iceberg compaction waiter dropped unexpectedly for si
• Sink not found: {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/fbfe18dbaeb8cb55. Report an issue: GitHub.