risingwavelabs/risingwave · error

pk-index sink epoch has no data files to commit

Error message

pk-index sink epoch {} has no data files to commit

What it means

pre_commit rejects an epoch when the aggregated writer reports contain neither data files nor delete files, yet at least one report had metadata. The exactly-once commit path requires the epoch to contribute at least one Iceberg file; an empty file set would produce a meaningless/invalid commit, so the meta node bails with this error.

Solutions

1. Check whether the epoch genuinely wrote data; if it is a no-op epoch, the writer should send metadata=None reports instead of empty metadata
2. Inspect writer logs and the serialized PbIcebergPkIndexSinkMetadata payloads for missing data_files/delete_files
3. Ensure writer and meta node versions are compatible (aggregation parses files with the same schema/format version)
4. If reproducible, file a bug with the reports; this state indicates an internal invariant violation

Example fix

// before (writer side)
send_report(metadata_with_no_files) // triggers bail in pre_commit
// after
if data_files.is_empty() && delete_files.is_empty() {
    send_report(PbIcebergPkIndexSinkMetadata { metadata: None, ..}); // signal no-op
} else {
    send_report(full_metadata);
}

// Writer-side: never send metadata reports with zero files
fn report_is_valid(r: &PbIcebergPkIndexSinkMetadata) -> bool {
    match &r.metadata {
        None => true, // explicit no-op
        Some(m) => !m.data_files.is_empty() || !m.delete_files.is_empty(),
    }
}
fn build_reports(...) -> Vec<PbIcebergPkIndexSinkMetadata> {
    if data_files.is_empty() && delete_files.is_empty() {
        vec![no_op_report()] // metadata: None
    } else {
        vec![full_report()]
    }
}

When it happens

Trigger: Calling pre_commit(prev_epoch, reports) where not all reports have metadata=None (so aggregation proceeds) but aggregate_reports(&reports)?.data_files and .delete_files are both empty.

Common situations: Sink writers flushed only internal state with no actual pk-index rows; a bug in the writer metadata serialization that emits metadata without file lists; schema/format mismatch between writer version and meta node causing files to be dropped during aggregation.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• iceberg pk-index sink commit failed for sink
• apply iceberg pk-index sink overwrite_files action
• backfill iceberg pk-index sink delete files failed…
• Failed to update iceberg table.
• Invalid commit metadata: empty payload

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/19e57aee2ea4bf63. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs:164

                .await
                .with_context(|| format!("drain recovered pending epoch for sink {}", sink_id))?;
        }

        Ok(coordinator)
    }

    pub async fn pre_commit(
        &mut self,
        prev_epoch: u64,
        reports: Vec<PbIcebergPkIndexSinkMetadata>,
    ) -> Result<()> {
        if reports.iter().all(|r| r.metadata.is_none()) {
            return Ok(());
        }

        let merged = aggregate_reports(&reports)?;
        if merged.data_files.is_empty() && merged.delete_files.is_empty() {
            bail!(
                "pk-index sink epoch {} has no data files to commit",
                prev_epoch
            );
        }
        let (merged, materialized_add_files) = self.backfill_delete_file_partitions(merged).await?;
        let merged = Arc::new(merged);

        let snapshot_id = FastAppendAction::generate_snapshot_id(&self.table);
        let blob = encode_pre_commit_state(&merged, snapshot_id)?;
        persist_pre_commit_metadata(&self.db, self.sink_id, prev_epoch, Some(blob), None).await?;

        self.waiting_commit = Some(EpochCommit {
            epoch: prev_epoch,
            merged,
            snapshot_id,
            materialized_add_files,
        });
        Ok(())

View on GitHub (pinned to 6469eb736d)