risingwavelabs/risingwave · error

compaction resolver PK column missing column_desc

Error message

compaction resolver PK column missing column_desc

What it means

When building PK data types for the Iceberg compaction resolver, each entry in `node.pk_columns` must carry a `column_desc` from which the `ColumnDesc` (and thus data type) is derived. A missing `column_desc` makes the PK type unknowable, so `new_boxed_executor` errors with this message.

Source

Thrown at src/stream/src/from_proto/iceberg_with_pk_index/compaction_resolver.rs:69

        let pk_indices = node
            .pk_columns
            .iter()
            .map(|column| column.data_file_index as usize)
            .collect::<Vec<_>>();
        if pk_indices.is_empty() {
            return Err(anyhow!("missing primary-key columns in compaction resolver").into());
        }

        let pk_data_types = node
            .pk_columns
            .iter()
            .map(|column| {
                column
                    .column_desc
                    .as_ref()
                    .map(ColumnDesc::from)
                    .map(|column| column.data_type)
                    .ok_or_else(|| anyhow!("compaction resolver PK column missing column_desc"))
            })
            .collect::<Result<Vec<_>, _>>()?;

        let barrier_receiver = params
            .local_barrier_manager
            .subscribe_barrier(params.actor_context.id);
        let local_barrier_manager = params.local_barrier_manager.clone();
        let meta_client = params.env.meta_client().ok_or_else(|| {
            anyhow!("meta client is required for iceberg pk-index compaction resolver")
        })?;
        let exec = CompactionResolverExecutor::new(
            params.actor_context,
            sink_id,
            iceberg_config,
            pk_indices,
            pk_data_types,
            params.config.developer.chunk_size,
            local_barrier_manager,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Regenerate the stream plan so each PK column includes its column_desc.
2. Upgrade frontend/meta together with the stream node to fix proto serialization mismatches.
3. Re-create the sink so the fragment carries complete column descriptors.

// before dispatching the plan
assert!(
    node.pk_columns.iter().all(|c| c.column_desc.is_some()),
    "every PK column must carry a column_desc"
);

fn pk_descs_present(node: &StreamNode) -> bool {
    node.pk_columns.iter().all(|c| c.column_desc.is_some())
}

When it happens

Trigger: A proto node whose `pk_columns[i].column_desc` is `None` during the `.map(...ok_or_else(...))` collection, i.e. partially serialized PK column metadata reaching the executor conversion.

Common situations: Proto serialization bugs dropping nested column_desc; schema evolution/version skew where old plans lack column descriptors; corrupted fragment graph during recovery or manual replay.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• {0}
• schema compilation error: {0}
• Can't find schema by id {}
• Field {} not found in our schema
• field {}'s type is incompatible RisingWave converted data ty

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/d13ade53a5e1294d. Report an issue: GitHub.