risingwavelabs/risingwave · error

pk-index sink pending row at epoch {} missing metadata blob

Error message

pk-index sink pending row at epoch {} missing metadata blob

What it means

On meta restart, `recovery` reads persisted sink-state rows ordered by epoch. A row in `Pending` state must contain the serialized pre-commit metadata blob; if the `metadata` column is NULL, the coordinator cannot reconstruct the pending commit and throws this error.

Source

Thrown at src/meta/src/manager/iceberg_pk_index_sink/coordinator.rs:404

) -> Result<(Option<u64>, Vec<EpochCommit>)> {
    fail::fail_point!("iceberg_v3_recovery_fail", |_| Err(anyhow::anyhow!(
        "injected: iceberg_v3_recovery_fail"
    )));
    let rows = list_sink_states_ordered_by_epoch(db, sink_id)
        .await
        .context("list pending sink states for pk-index sink recovery")?;

    let mut prev_committed_epoch = None;
    let mut pending = Vec::new();
    let mut aborted_epochs = Vec::new();
    for (epoch, state, metadata, _schema_change) in rows {
        match state {
            SinkState::Committed => {
                prev_committed_epoch = Some(epoch);
            }
            SinkState::Pending => {
                let blob = metadata.ok_or_else(|| {
                    anyhow!(
                        "pk-index sink pending row at epoch {} missing metadata blob",
                        epoch
                    )
                })?;
                let (merged, snapshot_id) = decode_pre_commit_state(&blob).with_context(|| {
                    format!("decode pk-index sink pre-commit state at epoch {}", epoch)
                })?;
                pending.push(EpochCommit {
                    epoch,
                    merged,
                    snapshot_id,
                    // Recovered from the persisted blob; the commit materializes files from
                    // `merged` once.
                    materialized_add_files: None,
                });
            }
            SinkState::Aborted => {
                // V3 doesn't produce Aborted rows; tolerate them defensively and drop them so they don't

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the meta store row (sink_states table) for the given sink_id/epoch to confirm the metadata column is NULL.
2. Remove the offending Pending row (or reset the sink) so recovery skips it and the epoch re-commits from fresh writer reports.
3. If reproducible, file/fix a bug ensuring the state row and metadata blob are written atomically.

Example fix

null

// before recovery, check rows are self-consistent
for row in rows {
    if row.state == SinkState::Pending {
        anyhow::ensure!(row.metadata.is_some(), "pending epoch {} has no metadata blob", row.epoch);
    }
}

fn pending_has_metadata(row: &SinkStateRow) -> bool {
    row.state != SinkState::Pending || row.metadata.is_some()
}

When it happens

Trigger: A `Pending` row was written without its metadata blob — e.g. partial/non-transactional write of the state row, manual DB pruning, or a bug that cleared metadata on state update.

Common situations: Meta store corrupted by manual intervention or a restore from a backup taken mid-write; schema migration that dropped the column's data; historical bug writing state before metadata.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• delete file {} missing referenced_data_file
• duplicate referenced data file {referenced} across pk-index
• no reports to aggregate for iceberg pk-index sink coordinato
• iceberg pk-index sink report missing metadata in aggregate_r
• compaction resolver sink {} task {} missing resolver task in

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/81ede7fb37a9db20. Report an issue: GitHub.