risingwavelabs/risingwave · error

Data file should not in task deletes

Error message

Data file should not in task deletes

What it means

While listing Iceberg scan tasks, the code iterates each task's delete files and dispatches on their content type. A `DataContentType::Data` file must never appear in `task.deletes` (deletes should only contain EqualityDeletes or PositionDeletes). If it does, the table metadata/scan planning produced an unexpected state, so the code bails as an internal invariant violation.

Source

Thrown at src/connector/src/source/iceberg/mod.rs:390

        let mut equality_delete_files = vec![];
        let mut equality_delete_files_set = HashSet::new();
        let mut equality_delete_ids = None;
        let mut scan_builder = table.scan().snapshot_id(snapshot_id).select_all();
        if predicate != IcebergPredicate::AlwaysTrue {
            scan_builder = scan_builder.with_filter(predicate.clone());
        }
        let scan = scan_builder.build()?;
        let file_scan_stream = scan.plan_files().await?;

        #[for_await]
        for task in file_scan_stream {
            let task: FileScanTask = task?;

            // Collect delete files for separate scan types, but keep task.deletes intact
            for delete_file in &task.deletes {
                match delete_file.file_type {
                    iceberg::spec::DataContentType::Data => {
                        bail!("Data file should not in task deletes");
                    }
                    iceberg::spec::DataContentType::EqualityDeletes => {
                        if equality_delete_files_set.insert(delete_file.file_path.clone()) {
                            if equality_delete_ids.is_none() {
                                equality_delete_ids = delete_file.equality_ids.clone();
                            } else if equality_delete_ids != delete_file.equality_ids {
                                bail!(
                                    "The schema of iceberg equality delete file must be consistent"
                                );
                            }
                            equality_delete_files.push(delete_file.to_file_scan_task(&task));
                        }
                    }
                    iceberg::spec::DataContentType::PositionDeletes => {
                        if position_delete_files_set.insert(delete_file.file_path.clone()) {
                            position_delete_files.push(delete_file.to_file_scan_task(&task));
                        }
                    }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the table metadata for the offending data file wrongly listed as a delete file.
2. Check the pinned iceberg-rust crate version for known bugs in FileScanTask delete classification; upgrade/downgrade accordingly.
3. Verify the snapshot/manifest being scanned is not corrupt by re-running with a different snapshot.
4. If reproducible, file/inspect how deletes are populated in list_scan_tasks_inner and add filtering or fix the dependency.

let bad = tasks.iter().flat_map(|t| t.deletes.iter()).any(|d| d.file_type == iceberg::spec::DataContentType::Data);
if bad { /* skip or regenerate scan tasks */ }

match list_scan_tasks(...) {
    Ok(r) => r,
    Err(e) if e.to_string().contains("Data file should not in task deletes") => {
        // re-plan with a different snapshot or report corrupt metadata
        ...
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: `list_scan_tasks`/`list_scan_tasks_inner` receiving a `FileScanTask` whose `deletes` vector contains a file with `file_type == DataContentType::Data` — indicative of a bug or corrupt/unexpected metadata from the iceberg-rust library version in use.

Common situations: Upgrading/changing the iceberg-rust dependency so delete files are classified differently; corrupt table metadata; a merge-on-read plan that incorrectly includes data files in the delete set.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• unsupported Iceberg file scan task type: {:?}
• compact_with_plan returned no result for a non-empty iceberg
• COW compaction must produce at most one table-scoped plan, g
• table {} not found
• register_table is not supported in mock catalog

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/ec7d9e1c3145605b. Report an issue: GitHub.