{"record":{"id":"fbfe18dbaeb8cb55","repo":"risingwavelabs/risingwave","slug":"iceberg-branch-disappeared-after-manifest-rewri","errorCode":null,"errorMessage":"Iceberg branch {} disappeared after manifest rewrite for sink {}","messagePattern":"Iceberg branch (.+?) disappeared after manifest rewrite for sink (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/manager/iceberg_compaction/gc.rs","lineNumber":441,"sourceCode":"        let estimated_output_manifest_count = plan.estimated_output_manifest_count;\n        let rewrite_paths = plan.rewrite_paths;\n        let txn = Transaction::new(&table);\n        let tx = txn\n            .rewrite_manifests()\n            .rewrite_if(Box::new(move |manifest| {\n                rewrite_paths.contains(&manifest.manifest_path)\n            }))\n            .cluster_by(Box::new(|_| \"risingwave-maintenance\".to_owned()))\n            .set_target_branch(branch.clone())\n            .apply(txn)\n            .map_err(|e| SinkError::Iceberg(e.into()))?;\n        let table = tx\n            .commit(catalog.as_ref())\n            .await\n            .map_err(|e| SinkError::Iceberg(e.into()))?;\n\n        let Some(rewritten_snapshot) = table.metadata().snapshot_for_ref(&branch) else {\n            return Err(anyhow!(\n                \"Iceberg branch {} disappeared after manifest rewrite for sink {}\",\n                branch,\n                sink_id\n            )\n            .into());\n        };\n        if rewritten_snapshot.snapshot_id() == current_snapshot_id {\n            tracing::warn!(\n                iceberg_component = \"manifest_maintenance\",\n                iceberg_operation = \"rewrite_manifests\",\n                table = %table_ident,\n                %sink_id,\n                %branch,\n                current_snapshot_id,\n                selected_manifest_count,\n                \"Manifest rewrite completed without creating a new snapshot\",\n            );\n            return Ok(());","sourceCodeStart":423,"sourceCodeEnd":459,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/manager/iceberg_compaction/gc.rs#L423-L459","documentation":"After committing an Iceberg manifest-rewrite transaction, the code looks up the branch's snapshot in the rewritten table metadata. If the branch reference is gone from the table metadata despite the commit succeeding, the internal invariant is broken (concurrent modification or catalog divergence), and GC fails with this error for the given sink.","triggerScenarios":"check_and_rewrite_manifests (invoked by perform_gc_operations) commits a manifest rewrite, then snapshot_for_ref(&branch) returns None because another process dropped/overwrote the branch ref between read and commit.","commonSituations":"Concurrent Iceberg GC/expire-snapshots jobs run by multiple meta processes; an external tool (Spark/Flink/Trino maintenance) removed the branch while RisingWave was rewriting manifests; misconfigured branch names changed outside RW.","solutions":["Re-run the GC/compaction cycle for the sink; the branch lookup will be re-attempted against fresh metadata.","Verify with the catalog that the branch still exists; recreate it if external tooling dropped it.","Ensure only one process performs manifest rewrite (locking/leader election) to avoid concurrent branch mutation.","Check other maintenance jobs' logs around the commit timestamp for branch removal."],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Before rewrite: confirm the branch exists and no other maintainer job runs concurrently\nlet snapshot = table.metadata().snapshot_for_ref(&branch)\n    .ok_or_else(|| anyhow!(\"branch {} missing before rewrite\", branch))?;","typeGuard":null,"tryCatchPattern":"match check_and_rewrite_manifests(...).await {\n    Err(e) if e.to_string().contains(\"disappeared after manifest rewrite\") => {\n        warn!(\"branch ref changed concurrently; re-running GC\");\n        retry_with_backoff(3, || check_and_rewrite_manifests(...)).await\n    }\n    other => other,\n}","preventionTips":["Ensure single-writer semantics for Iceberg maintenance (leader election / table-level locking).","Do not run external expire-snapshots/branch-drop jobs during RW GC windows.","Verify branch names used by RW sinks are owned and immutable outside RW."],"tags":["rust","meta-service","iceberg","gc","concurrency"],"backgroundTag":"unexpected-response-shape","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}