{"record":{"id":"142041813f08342e","repo":"risingwavelabs/risingwave","slug":"backfill-iceberg-pk-index-sink-delete-files-failed","errorCode":null,"errorMessage":"backfill iceberg pk-index sink delete files failed, unresolved referenced data files: {:?}","messagePattern":"backfill iceberg pk-index sink delete files failed, unresolved referenced data files: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs","lineNumber":280,"sourceCode":"            .into_iter()\n            .map(|f| f.try_into(merged.partition_spec_id, &partition_type, schema))\n            .try_collect::<Vec<DataFile>>()?;\n\n        if !delete_files.is_empty() {\n            let mut pending = pending_delete_files_by_referenced(&mut delete_files)?;\n            for f in &data_files {\n                if let Some(delete_file) = pending.remove(f.file_path()) {\n                    delete_file.set_partition(f.partition().clone());\n                    if pending.is_empty() {\n                        break;\n                    }\n                }\n            }\n            if !pending.is_empty() {\n                probe_committed_data_files(&self.table, &mut pending).await?;\n            }\n            if !pending.is_empty() {\n                anyhow::bail!(\n                    \"backfill iceberg pk-index sink delete files failed, unresolved referenced data files: {:?}\",\n                    pending.keys().collect::<Vec<_>>()\n                );\n            }\n        }\n\n        let serialized_delete_files = delete_files\n            .iter()\n            .cloned()\n            .map(|f| SerializedDataFile::try_from(f, &partition_type, format_version))\n            .try_collect()?;\n\n        let merged = IcebergPkIndexSinkAggResult {\n            schema_id: merged.schema_id,\n            partition_spec_id: merged.partition_spec_id,\n            data_files: merged.data_files,\n            delete_files: serialized_delete_files,\n            overwrite_files: merged.overwrite_files,","sourceCodeStart":262,"sourceCodeEnd":298,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs#L262-L298","documentation":"During pre_commit on a partitioned table, each delete file must inherit the partition values of the data file it references. The coordinator resolves partitions from this epoch's data files and then probes the currently committed snapshot via probe_committed_data_files; if some referenced data files still cannot be found, the backfill fails because committing a delete file with unresolved partition would be invalid.","triggerScenarios":"pre_commit -> backfill_delete_file_partitions on a partitioned table where a delete file's referenced_data_file path is present in neither the epoch's writer data files nor the committed snapshot's data files.","commonSituations":"The referenced data file was removed by concurrent compaction/expiration of snapshots on the Iceberg table, table history was rewritten externally, or writer metadata references files from an old snapshot that has since been replaced; also possible with wrong branch/table pointing to a different snapshot lineage.","solutions":["Check the listed data file paths against the table's current snapshot manifest data to confirm they were removed","Avoid external snapshot expiration/rewrite racing the sink: pause Iceberg maintenance (expire_snapshots, rewrite_data_files) during backfill/commits","Verify the sink writes to a dedicated branch to isolate from external maintenance","Check writer side for stale references: reports should only reference files from recent epochs","If reproducible with no external interference, file a bug with the unresolved paths"],"exampleFix":"// before\nexternal_job.expire_snapshots(older_than = 1h) // removes files still referenced by pending epoch\n// after\nexternal_job.expire_snapshots(older_than = 24h) // keep files referenced by in-flight epochs\n// and write sink to dedicated branch:\niceberg.write_mode = \"dedicated_branch\"","handlingStrategy":"validation","validationCode":"// Before pre_commit on partitioned tables, ensure referenced files still exist in the snapshot\nasync fn referenced_files_resolvable(table: &Table, reports: &[PbIcebergPkIndexSinkMetadata]) -> bool {\n    let referenced: HashSet<String> = collect_referenced_data_files(reports);\n    let snapshot_files = current_snapshot_data_file_paths(table);\n    referenced.iter().all(|f| snapshot_files.contains(f))\n}","typeGuard":null,"tryCatchPattern":"match coordinator.pre_commit(epoch, reports).await {\n    Err(e) if e.to_string().contains(\"unresolved referenced data files\") => {\n        // pause external Iceberg maintenance and re-drive the epoch from pending state\n        pause_snapshot_expiration();\n        retry_pre_commit(epoch, reports).await\n    }\n    other => other,\n}","preventionTips":["Never expire snapshots/rewrite data files concurrently with sink commits on the same table","Write sink output to a dedicated branch isolated from external Iceberg maintenance","Keep expiration retention longer than the worst-case pre_commit->commit window","Compare unresolved paths against the table's manifests to confirm external removal vs writer bug"],"tags":["iceberg","delete-files","partition","commit"],"backgroundTag":"resource-not-found","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}