risingwavelabs/risingwave · error

backfill iceberg pk-index sink delete files failed…

Error message

backfill iceberg pk-index sink delete files failed, unresolved referenced data files: {:?}

What it means

During pre_commit on a partitioned table, each delete file must inherit the partition values of the data file it references. The coordinator resolves partitions from this epoch's data files and then probes the currently committed snapshot via probe_committed_data_files; if some referenced data files still cannot be found, the backfill fails because committing a delete file with unresolved partition would be invalid.

Solutions

1. Check the listed data file paths against the table's current snapshot manifest data to confirm they were removed
2. Avoid external snapshot expiration/rewrite racing the sink: pause Iceberg maintenance (expire_snapshots, rewrite_data_files) during backfill/commits
3. Verify the sink writes to a dedicated branch to isolate from external maintenance
4. Check writer side for stale references: reports should only reference files from recent epochs
5. If reproducible with no external interference, file a bug with the unresolved paths

Example fix

// before
external_job.expire_snapshots(older_than = 1h) // removes files still referenced by pending epoch
// after
external_job.expire_snapshots(older_than = 24h) // keep files referenced by in-flight epochs
// and write sink to dedicated branch:
iceberg.write_mode = "dedicated_branch"

// Before pre_commit on partitioned tables, ensure referenced files still exist in the snapshot
async fn referenced_files_resolvable(table: &Table, reports: &[PbIcebergPkIndexSinkMetadata]) -> bool {
    let referenced: HashSet<String> = collect_referenced_data_files(reports);
    let snapshot_files = current_snapshot_data_file_paths(table);
    referenced.iter().all(|f| snapshot_files.contains(f))
}

match coordinator.pre_commit(epoch, reports).await {
    Err(e) if e.to_string().contains("unresolved referenced data files") => {
        // pause external Iceberg maintenance and re-drive the epoch from pending state
        pause_snapshot_expiration();
        retry_pre_commit(epoch, reports).await
    }
    other => other,
}

When it happens

Trigger: pre_commit -> backfill_delete_file_partitions on a partitioned table where a delete file's referenced_data_file path is present in neither the epoch's writer data files nor the committed snapshot's data files.

Common situations: The referenced data file was removed by concurrent compaction/expiration of snapshots on the Iceberg table, table history was rewritten externally, or writer metadata references files from an old snapshot that has since been replaced; also possible with wrong branch/table pointing to a different snapshot lineage.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• apply iceberg pk-index sink overwrite_files action
• failed to add partition columns
• Failed to update iceberg table.
• iceberg pk-index sink commit failed for sink
• Invalid commit metadata: empty payload

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/142041813f08342e. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs:280

            .into_iter()
            .map(|f| f.try_into(merged.partition_spec_id, &partition_type, schema))
            .try_collect::<Vec<DataFile>>()?;

        if !delete_files.is_empty() {
            let mut pending = pending_delete_files_by_referenced(&mut delete_files)?;
            for f in &data_files {
                if let Some(delete_file) = pending.remove(f.file_path()) {
                    delete_file.set_partition(f.partition().clone());
                    if pending.is_empty() {
                        break;
                    }
                }
            }
            if !pending.is_empty() {
                probe_committed_data_files(&self.table, &mut pending).await?;
            }
            if !pending.is_empty() {
                anyhow::bail!(
                    "backfill iceberg pk-index sink delete files failed, unresolved referenced data files: {:?}",
                    pending.keys().collect::<Vec<_>>()
                );
            }
        }

        let serialized_delete_files = delete_files
            .iter()
            .cloned()
            .map(|f| SerializedDataFile::try_from(f, &partition_type, format_version))
            .try_collect()?;

        let merged = IcebergPkIndexSinkAggResult {
            schema_id: merged.schema_id,
            partition_spec_id: merged.partition_spec_id,
            data_files: merged.data_files,
            delete_files: serialized_delete_files,
            overwrite_files: merged.overwrite_files,

View on GitHub (pinned to 6469eb736d)