risingwavelabs/risingwave · error · SinkError::Iceberg

Invalid commit metadata: missing snapshot_id

Error message

Invalid commit metadata: missing snapshot_id

What it means

After popping the last payload element, commit_data treats it as the little-endian snapshot_id bytes. If the payload list was empty at that point (only reachable when the earlier is_empty guard passed but pop still fails — effectively an invariant race), it returns this error. The snapshot id is mandatory to create the new Iceberg snapshot.

Source

Thrown at src/connector/src/sink/iceberg/commit.rs:498

                sink_id = %self.sink_id,
                table = %self.table.identifier(),
                epoch,
                "iceberg_sink_commit_skipped_empty_metadata",
            );
            return Ok(());
        }

        // Deserialize commit metadata
        let mut payload = deserialize_metadata(commit_metadata);
        if payload.is_empty() {
            return Err(SinkError::Iceberg(anyhow!(
                "Invalid commit metadata: empty payload"
            )));
        }

        // Last element is snapshot_id
        let snapshot_id_bytes = payload.pop().ok_or_else(|| {
            SinkError::Iceberg(anyhow!("Invalid commit metadata: missing snapshot_id"))
        })?;
        let snapshot_id = i64::from_le_bytes(
            snapshot_id_bytes
                .try_into()
                .map_err(|_| SinkError::Iceberg(anyhow!("Invalid snapshot id bytes")))?,
        );

        // Remaining elements are write_results
        let write_results = payload
            .into_iter()
            .map(|p| IcebergCommitResult::try_from_serialized_bytes(&p))
            .collect::<Result<Vec<_>>>()?;

        let snapshot_committed = self.is_snapshot_id_in_iceberg(snapshot_id).await?;

        if snapshot_committed {
            tracing::info!(
                iceberg_component = "sink_committer",

View on GitHub (pinned to 6469eb736d)

Solutions

1. Ensure the metadata producer always appends snapshot_id.to_le_bytes() as the final payload element
2. Align the writer's serialization format with commit_data's expected layout
3. Regenerate the commit metadata by restarting the sink epoch

Example fix

// before (payload without snapshot id)
payload = [write_result_1_bytes]
// after
payload = [write_result_1_bytes, snapshot_id.to_le_bytes()]

fn payload_has_snapshot_id(payload: &[Vec<u8>]) -> bool {
    payload.len() >= 1 && payload.last().map_or(false, |b| b.len() == 8)
}

fn as_le_i64(bytes: &[u8]) -> Option<i64> {
    i64::from_le_bytes(bytes.try_into().ok()?).into()
}

let Some(snapshot_id_bytes) = payload.pop() else {
    return Err(SinkError::Iceberg(anyhow!("missing snapshot_id")));
};

When it happens

Trigger: commit_data with a payload that has no trailing snapshot_id element — a metadata blob assembled without appending the i64 LE snapshot id.

Common situations: Custom or outdated code paths assembling commit metadata without the snapshot id suffix; corrupted state where the last element was dropped.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• iceberg sink metadata should have schema_id
• iceberg sink metadata should have partition_spec_id
• iceberg sink metadata should have data_files object
• Invalid commit metadata: empty payload
• query_epoch not set in distributed lookup join

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/f96191e96233ae36. Report an issue: GitHub.