risingwavelabs/risingwave · error

empty sink metadata

Error message

empty sink metadata

What it means

A `CommitRequest` arrived from a writer handle but its `metadata` field is `None`. Sink metadata (e.g. the external sink's target/status info needed for the commit) is required for every commit, so the manager rejects the request with this error. It indicates the writer sent a malformed or metadata-less commit message over the gRPC coordination stream.

Source

Thrown at src/meta/src/manager/sink_coordination/coordinator_worker.rs:402

    async fn next_event(&mut self) -> anyhow::Result<(HandleId, CoordinationHandleManagerEvent)> {
        select! {
            handle = self.request_rx.recv() => {
                let handle = handle.ok_or_else(|| anyhow!("end of writer request stream"))?;
                if handle.param() != &self.param {
                    warn!(prev_param = ?self.param, new_param = ?handle.param(), "sink param mismatch");
                }
                let handle_id = self.next_handle_id;
                self.next_handle_id += 1;
                self.writer_handles.insert(handle_id, handle);
                Ok((handle_id, CoordinationHandleManagerEvent::NewHandle))
            }
            result = Self::next_request_inner(&mut self.writer_handles) => {
                let (handle_id, request) = result?;
                let event = match request {
                    coordinate_request::Msg::CommitRequest(request) => {
                        CoordinationHandleManagerEvent::CommitRequest {
                            epoch: request.epoch,
                            metadata: request.metadata.ok_or_else(|| anyhow!("empty sink metadata"))?,
                            schema_change: request.schema_change,
                        }
                    }
                    coordinate_request::Msg::AlignInitialEpochRequest(epoch) => {
                        CoordinationHandleManagerEvent::AlignInitialEpoch(epoch)
                    }
                    coordinate_request::Msg::UpdateVnodeRequest(_) => {
                        CoordinationHandleManagerEvent::UpdateVnodeBitmap
                    }
                    coordinate_request::Msg::Stop(_) => {
                        CoordinationHandleManagerEvent::Stop
                    }
                    coordinate_request::Msg::StartRequest(_) => {
                        unreachable!("should have been handled");
                    }
                };
                Ok((handle_id, event))
            }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check the writer side (stream executor) to ensure `sink_metadata` is populated before sending `CommitRequest`.
2. Look for recent protobuf changes to `coordinate_request` and align binaries across the cluster (rolling upgrade skew).
3. Verify the connector's sink metadata serialization path for the failing sink type (e.g. Kafka/Iceberg).
4. Add a validation at the writer so a commit without metadata fails fast locally instead of reaching the meta node.

Example fix

// before
let msg = coordinate_request::Msg::CommitRequest(CommitRequest {
    epoch,
    metadata: None,
    schema_change: None,
});

// after
let msg = coordinate_request::Msg::CommitRequest(CommitRequest {
    epoch,
    metadata: Some(build_sink_metadata(&sink).expect("sink metadata required for commit")),
    schema_change: None,
});

// On the writer side, refuse to send commits without metadata
let metadata = build_sink_metadata(&sink)
    .ok_or_else(|| anyhow!("sink metadata must be set before commit"))?;
let msg = CommitRequest { epoch, metadata: Some(metadata), schema_change };
anyhow::ensure!(msg.metadata.is_some(), "CommitRequest requires metadata");

fn has_metadata(req: &CommitRequest) -> bool {
    req.metadata.is_some()
}

if let Err(e) = manager.next_event().await {
    if e.to_string().contains("empty sink metadata") {
        error!("malformed CommitRequest from writer; check version skew and connector metadata serialization");
    }
    return Err(e);
}

When it happens

Trigger: A stream writer builds `coordinate_request::Msg::CommitRequest` without setting `metadata` — e.g. after a protobuf schema change where the field was renamed/newer writers omit it, or a bug in the commit path that forgets to populate `sink_metadata`.

Common situations: Version skew between frontend/stream compute and meta node after a protobuf field change; a connector whose sink metadata serialization fails and silently produces `None`; hand-rolled test clients sending incomplete commit requests.

Understand the failure class

Background: "must not be empty", "cannot be empty" — required-field validation errors across open-source libraries — this error's family across 41 libraries.

Related errors

• query_epoch not set in distributed lookup join
• empty commit metadata
• Failed to decode prost: field not found `{}`
• Invalid datetime: seconds: {secs}, nanoseconds: {nsecs}
• Can't find message proto {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/7c7b4d5ae61c4308. Report an issue: GitHub.