{"record":{"id":"46c426701336766e","repo":"risingwavelabs/risingwave","slug":"duplicate-referenced-data-file-referenced-across","errorCode":null,"errorMessage":"duplicate referenced data file {referenced} across pk-index delete files","messagePattern":"duplicate referenced data file (.+?) across pk-index delete files","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs","lineNumber":318,"sourceCode":"\n        // `add_data_files` order in `commit_one_epoch` is data files followed by\n        // delete files; keep that order here.\n        let mut materialized_add_files = data_files;\n        materialized_add_files.extend(delete_files);\n        Ok((merged, Some(materialized_add_files)))\n    }\n}\n\npub fn pending_delete_files_by_referenced(\n    delete_files: &mut [DataFile],\n) -> Result<HashMap<String, &mut DataFile>> {\n    let mut pending: HashMap<String, &mut DataFile> = HashMap::with_capacity(delete_files.len());\n    for f in delete_files.iter_mut() {\n        let referenced = f.referenced_data_file().ok_or_else(|| {\n            anyhow::anyhow!(\"delete file {} missing referenced_data_file\", f.file_path())\n        })?;\n        if pending.contains_key(&referenced) {\n            anyhow::bail!(\n                \"duplicate referenced data file {referenced} across pk-index delete files\"\n            );\n        }\n        pending.insert(referenced, f);\n    }\n    Ok(pending)\n}\n\n/// Scan the table's current-snapshot data manifests, resolving pending delete-file\n/// partitions as their referenced data files are found. Stops as soon as every pending\n/// delete file has been resolved. Data-file references not present in the snapshot are\n/// left in `pending` for the caller to report.\nasync fn probe_committed_data_files(\n    table: &Table,\n    pending: &mut HashMap<String, &mut DataFile>,\n) -> Result<()> {\n    let Some(snapshot) = table.metadata().current_snapshot() else {\n        return Ok(());","sourceCodeStart":300,"sourceCodeEnd":336,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs#L300-L336","documentation":"`pending_delete_files_by_referenced` requires each delete file to reference a distinct data file, since the map is keyed by the referenced data-file path. Two delete files pointing at the same referenced data file make the partition backfill ambiguous, so it bails with this message.","triggerScenarios":"Two or more position-delete files in a single pre-commit aggregation report the same `referenced_data_file` — e.g. duplicate reports from a retried PositionDeleteMerger, or duplicate rows aggregated across writer retries.","commonSituations":"Writer/merger tasks re-executed after a retry produce duplicate delete files within one epoch's commit reports; a bug in report deduplication in `aggregate_reports`.","solutions":["Deduplicate delete files by (file path or referenced data file) before calling `backfill_delete_file_partitions`.","Investigate why the PositionDeleteMerger emitted two delete files for the same referenced data file in one epoch (task retry/panic replay).","Ensure each epoch's commit is only aggregated once: check `prev_committed_epoch` bookkeeping so retried epochs don't double-report."],"exampleFix":"// before\nlet delete_files = merged.delete_files;\npending_delete_files_by_referenced(&mut delete_files)?;\n// after\nlet mut seen = HashSet::new();\ndelete_files.retain(|f| seen.insert(f.file_path().to_string()));\npending_delete_files_by_referenced(&mut delete_files)?;","handlingStrategy":"validation","validationCode":"let mut seen = HashSet::new();\nfor f in delete_files {\n    let r = f.referenced_data_file().unwrap();\n    anyhow::ensure!(seen.insert(r.to_string()), \"duplicate referenced data file {r}\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Deduplicate delete files by file path before aggregation.","Make merger tasks idempotent so retries don't emit duplicate delete files.","Ensure each epoch's reports are aggregated exactly once."],"tags":["iceberg","duplicate-data","internal-invariant-violation"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}