duplicate referenced data file {referenced} across pk-index

Error message

duplicate referenced data file {referenced} across pk-index delete files

What it means

`pending_delete_files_by_referenced` requires each delete file to reference a distinct data file, since the map is keyed by the referenced data-file path. Two delete files pointing at the same referenced data file make the partition backfill ambiguous, so it bails with this message.

Source

Thrown at src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs:318

        // `add_data_files` order in `commit_one_epoch` is data files followed by
        // delete files; keep that order here.
        let mut materialized_add_files = data_files;
        materialized_add_files.extend(delete_files);
        Ok((merged, Some(materialized_add_files)))
    }
}

pub fn pending_delete_files_by_referenced(
    delete_files: &mut [DataFile],
) -> Result<HashMap<String, &mut DataFile>> {
    let mut pending: HashMap<String, &mut DataFile> = HashMap::with_capacity(delete_files.len());
    for f in delete_files.iter_mut() {
        let referenced = f.referenced_data_file().ok_or_else(|| {
            anyhow::anyhow!("delete file {} missing referenced_data_file", f.file_path())
        })?;
        if pending.contains_key(&referenced) {
            anyhow::bail!(
                "duplicate referenced data file {referenced} across pk-index delete files"
            );
        }
        pending.insert(referenced, f);
    }
    Ok(pending)
}

/// Scan the table's current-snapshot data manifests, resolving pending delete-file
/// partitions as their referenced data files are found. Stops as soon as every pending
/// delete file has been resolved. Data-file references not present in the snapshot are
/// left in `pending` for the caller to report.
async fn probe_committed_data_files(
    table: &Table,
    pending: &mut HashMap<String, &mut DataFile>,
) -> Result<()> {
    let Some(snapshot) = table.metadata().current_snapshot() else {
        return Ok(());

View on GitHub (pinned to 6469eb736d)

Solutions

Deduplicate delete files by (file path or referenced data file) before calling `backfill_delete_file_partitions`.
Investigate why the PositionDeleteMerger emitted two delete files for the same referenced data file in one epoch (task retry/panic replay).
Ensure each epoch's commit is only aggregated once: check `prev_committed_epoch` bookkeeping so retried epochs don't double-report.

Example fix

// before
let delete_files = merged.delete_files;
pending_delete_files_by_referenced(&mut delete_files)?;
// after
let mut seen = HashSet::new();
delete_files.retain(|f| seen.insert(f.file_path().to_string()));
pending_delete_files_by_referenced(&mut delete_files)?;

Defensive patterns

Strategy: validation

Validate before calling

let mut seen = HashSet::new();
for f in delete_files {
    let r = f.referenced_data_file().unwrap();
    anyhow::ensure!(seen.insert(r.to_string()), "duplicate referenced data file {r}");
}

Prevention

Deduplicate delete files by file path before aggregation.
Make merger tasks idempotent so retries don't emit duplicate delete files.
Ensure each epoch's reports are aggregated exactly once.

When it happens

Trigger: Two or more position-delete files in a single pre-commit aggregation report the same `referenced_data_file` — e.g. duplicate reports from a retried PositionDeleteMerger, or duplicate rows aggregated across writer retries.

Common situations: Writer/merger tasks re-executed after a retry produce duplicate delete files within one epoch's commit reports; a bug in report deduplication in `aggregate_reports`.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/46c426701336766e. Report an issue: GitHub.