{"record":{"id":"afd5a81526656f0e","repo":"risingwavelabs/risingwave","slug":"delete-file-missing-referenced-data-file","errorCode":null,"errorMessage":"delete file {} missing referenced_data_file","messagePattern":"delete file (.+?) missing referenced_data_file","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs","lineNumber":315,"sourceCode":"            delete_files: serialized_delete_files,\n            overwrite_files: merged.overwrite_files,\n        };\n\n        // `add_data_files` order in `commit_one_epoch` is data files followed by\n        // delete files; keep that order here.\n        let mut materialized_add_files = data_files;\n        materialized_add_files.extend(delete_files);\n        Ok((merged, Some(materialized_add_files)))\n    }\n}\n\npub fn pending_delete_files_by_referenced(\n    delete_files: &mut [DataFile],\n) -> Result<HashMap<String, &mut DataFile>> {\n    let mut pending: HashMap<String, &mut DataFile> = HashMap::with_capacity(delete_files.len());\n    for f in delete_files.iter_mut() {\n        let referenced = f.referenced_data_file().ok_or_else(|| {\n            anyhow::anyhow!(\"delete file {} missing referenced_data_file\", f.file_path())\n        })?;\n        if pending.contains_key(&referenced) {\n            anyhow::bail!(\n                \"duplicate referenced data file {referenced} across pk-index delete files\"\n            );\n        }\n        pending.insert(referenced, f);\n    }\n    Ok(pending)\n}\n\n/// Scan the table's current-snapshot data manifests, resolving pending delete-file\n/// partitions as their referenced data files are found. Stops as soon as every pending\n/// delete file has been resolved. Data-file references not present in the snapshot are\n/// left in `pending` for the caller to report.\nasync fn probe_committed_data_files(\n    table: &Table,\n    pending: &mut HashMap<String, &mut DataFile>,","sourceCodeStart":297,"sourceCodeEnd":333,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs#L297-L333","documentation":"During delete-file partition backfill, the coordinator builds a map from each delete file to the data file it references. Each Iceberg delete file produced by the pk-index position-delete merger must carry a `referenced_data_file` property naming the data file it belongs to. If a delete file lacks that property, `pending_delete_files_by_referenced` cannot determine which partition to backfill it into and throws this error naming the offending file path.","triggerScenarios":"A SerializedDataFile for a delete file was persisted/decoded without the `referenced-data-file` property set (e.g. writer bug, older wire format, hand-edited metadata), then passed into `backfill_delete_file_partitions` during pre-commit or recovery.","commonSituations":"Recovering a sink whose pending rows were written by an older RisingWave version that did not stamp `referenced_data_file`; a connector/writer bug that dropped the property during serialization; manually repaired Iceberg metadata tables.","solutions":["Fix or re-run the upstream writer (PositionDeleteMerger) so every delete file is created with the `referenced-data-file` property populated.","Check the RisingWave version that produced the pending sink-state rows; upgrade/restart the meta node so both writer and coordinator versions match.","If a bad row is already persisted, delete/clean the sink state for the affected epoch and let the epoch re-commit from the writers' reports."],"exampleFix":"// before: delete file created without reference\nDataFile::PositionDeletesBuilder::new(...).build()?;\n// after\nbuilder.set_referenced_data_file(data_file_path.to_string());\nlet df = builder.build()?;","handlingStrategy":"validation","validationCode":"for f in delete_files {\n    anyhow::ensure!(f.referenced_data_file().is_some(), \"delete file {} missing referenced_data_file\", f.file_path());\n}","typeGuard":"fn has_referenced_data_file(f: &DataFile) -> bool { f.referenced_data_file().is_some() }","tryCatchPattern":null,"preventionTips":["Always set `referenced-data-file` when building position-delete files in the writer.","Add a unit test asserting every serialized delete file round-trips with its reference intact.","Pin writer and meta node versions together to avoid old-format pending rows."],"tags":["iceberg","internal-invariant-violation","delete-file","recovery"],"backgroundTag":"missing-required-argument","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}