{"record":{"id":"c0ebc8662c43cd21","repo":"risingwavelabs/risingwave","slug":"the-schema-of-iceberg-equality-delete-file-must-be","errorCode":null,"errorMessage":"The schema of iceberg equality delete file must be consistent","messagePattern":"The schema of iceberg equality delete file must be consistent","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"src/connector/src/source/iceberg/mod.rs","lineNumber":397,"sourceCode":"        let scan = scan_builder.build()?;\n        let file_scan_stream = scan.plan_files().await?;\n\n        #[for_await]\n        for task in file_scan_stream {\n            let task: FileScanTask = task?;\n\n            // Collect delete files for separate scan types, but keep task.deletes intact\n            for delete_file in &task.deletes {\n                match delete_file.file_type {\n                    iceberg::spec::DataContentType::Data => {\n                        bail!(\"Data file should not in task deletes\");\n                    }\n                    iceberg::spec::DataContentType::EqualityDeletes => {\n                        if equality_delete_files_set.insert(delete_file.file_path.clone()) {\n                            if equality_delete_ids.is_none() {\n                                equality_delete_ids = delete_file.equality_ids.clone();\n                            } else if equality_delete_ids != delete_file.equality_ids {\n                                bail!(\n                                    \"The schema of iceberg equality delete file must be consistent\"\n                                );\n                            }\n                            equality_delete_files.push(delete_file.to_file_scan_task(&task));\n                        }\n                    }\n                    iceberg::spec::DataContentType::PositionDeletes => {\n                        if position_delete_files_set.insert(delete_file.file_path.clone()) {\n                            position_delete_files.push(delete_file.to_file_scan_task(&task));\n                        }\n                    }\n                }\n            }\n\n            // Top-level scan tasks always represent data files. Keep their delete\n            // descriptors intact so the SDK reader can apply them when requested.\n            data_files.push(task);\n        }","sourceCodeStart":379,"sourceCodeEnd":415,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/source/iceberg/mod.rs#L379-L415","documentation":"When multiple equality-delete files are collected for a scan, all of them must use the same equality field ids (i.e., the same delete schema). If a second equality delete file has different `equality_ids` than the ones already collected, RisingWave cannot apply them consistently and bails with this error.","triggerScenarios":"`list_scan_tasks`/`list_scan_tasks_inner` encountering two or more equality delete files in a scan task's deletes where their `equality_ids` differ — e.g. a table written by different writers with different equality-delete schemas or evolved schema between delete writes.","commonSituations":"Tables written by multiple engines/writers (Spark, Flink, Trino) with mismatched equality-delete field sets; Iceberg schema evolution changing the delete columns between writes; copy/merge operations that mixed delete schemas.","solutions":["Rewrite the affected data with a compaction/rewrite job (e.g. Spark `rewrite_data_files` / `rewrite_delete_files`) so only one equality-delete schema remains.","Align writers so all use the same equality fields for deletes.","Check schema evolution history; avoid changing delete key columns, or compact deletes after evolving the schema.","Use copy-on-write mode for deletes if mixed schemas are unavoidable."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"let ids: HashSet<_> = tasks.iter().flat_map(|t| t.deletes.iter())\n    .filter(|d| d.file_type == iceberg::spec::DataContentType::EqualityDeletes)\n    .map(|d| d.equality_ids.clone()).collect();\nif ids.len() > 1 { /* compact deletes or reject scan */ }","typeGuard":null,"tryCatchPattern":"match list_scan_tasks(...) {\n    Err(e) if e.to_string().contains(\"equality delete file must be consistent\") => {\n        // trigger compaction/rewrite or scan without MoR deletes\n    }\n    r => r,\n}","preventionTips":["Use a single writer with a fixed equality-delete schema","Compact delete files after schema evolution","Prefer copy-on-write when multiple engines write to the table"],"tags":["rust","iceberg","equality-delete","schema"],"backgroundTag":"schema-validation-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}