risingwavelabs/risingwave · error · anyhow::Error

Partition source column does not exist in schema: {}

Error message

Partition source column does not exist in schema: {}

What it means

When the sink defines partitioning, each partition column must exist in the sink schema so a partition field (with source_id and transform) can be built. If a configured partition column name does not match any schema column, creation fails with this error naming the missing column.

Source

Thrown at src/connector/src/sink/iceberg/create_table.rs:196

    };

    let partition_spec = match &config.partition_by {
        Some(partition_by) => {
            let mut partition_fields = Vec::<UnboundPartitionField>::new();
            for (i, (column, transform)) in parse_partition_by_exprs(partition_by.clone())?
                .into_iter()
                .enumerate()
            {
                match iceberg_schema.field_id_by_name(&column) {
                    Some(id) => partition_fields.push(
                        UnboundPartitionField::builder()
                            .source_id(id)
                            .transform(transform)
                            .name(format!("_p_{}", column))
                            .field_id(PARTITION_DATA_ID_START + i as i32)
                            .build(),
                    ),
                    None => bail!(format!(
                        "Partition source column does not exist in schema: {}",
                        column
                    )),
                };
            }
            Some(
                UnboundPartitionSpec::builder()
                    .with_spec_id(0)
                    .add_partition_fields(partition_fields)
                    .map_err(|e| SinkError::Iceberg(anyhow!(e)))
                    .context("failed to add partition columns")?
                    .build(),
            )
        }
        None => None,
    };

    let sort_order = match &config.order_key {

View on GitHub (pinned to 6469eb736d)

Solutions

  1. Correct the partition column name in `partition.by` to exactly match a sink column.
  2. Ensure the column exists in the materialized view/sink output (add it if it was dropped).
  3. Recreate the sink after schema changes to refresh the partition config.
  4. Check column name casing matches the schema exactly.

Example fix

// before
WITH (connector='iceberg', partition.by='event_day') -- column doesn't exist
// after: add the column or fix the name
CREATE MATERIALIZED VIEW mv AS SELECT date_trunc('day', ts) AS event_day, ... ;
WITH (connector='iceberg', partition.by='event_day')
Defensive patterns

Strategy: validation

Validate before calling

// Ensure every partition column exists in the sink schema
for col in partition_columns {
    if !schema.fields().iter().any(|f| f.name().field_name == col) {
        eprintln!("partition column {col} missing from sink schema");
    }
}

Try / catch

match result {
    Err(e) if e.to_string().contains("Partition source column does not exist") => {
        // fix partition.by names or add the column to the MV
    }
    other => other?,
}

Prevention

When it happens

Trigger: create_table_if_not_exists_impl looks up each name from the sink's partition config (column) in the schema's name-to-id map; the lookup returns None and the code bails — e.g. `partition.by = 'event_day'` where no column named event_day exists in the sink.

Common situations: Partition column renamed or dropped from the MV after the sink was defined; typo in partition.by; partitioning on a derived/alias column not included in the sink schema; case-sensitivity mismatch.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/7b96138674fb16b9. Report an issue: GitHub.