risingwavelabs/risingwave · error · SinkError::Iceberg

Failed to convert Arrow type to Iceberg type

Error message

Failed to convert Arrow type to Iceberg type

What it means

commit_schema_change_impl converts each added column's Arrow DataType to an Iceberg Type via `iceberg::arrow::arrow_type_to_type`; some Arrow types have no Iceberg representation, and the conversion returns an error which is wrapped with this context. The schema evolution commit is aborted before any metadata update.

Solutions

1. Change the added column's type to an Iceberg-compatible one (e.g., timestamp, decimal, string, int/float/bool, supported nested types).
2. Cast the column to a supported type in the upstream/stream plan before it reaches the sink.
3. Drop and re-add the column with a supported type, then let schema evolution re-run.
4. Upgrade iceberg-rust/RisingWave if a newer version supports the Arrow type.

Example fix

// before: adding unsupported type directly
new_fields.push(AddColumn::optional(&field.name, iceberg_type));
// after: cast unsupported types in the upstream plan, e.g. to string
// ALTER TABLE t ADD COLUMN flags jsonb  ->  cast in sink plan:
// flags::varchar, then iceberg_type = arrow_type_to_type(Utf8)?

// Check the column type is Iceberg-representable before adding it upstream:
fn is_iceberg_comparable(dt: &arrow::datatypes::DataType) -> bool {
    use arrow::datatypes::DataType::*;
    matches!(dt, Null | Boolean | Int8 | Int16 | Int32 | Int64 | Float32 | Float64 | Utf8 | Binary | Date32 | Timestamp(_, _) | Decimal128(_, _) | _) // tighten per iceberg::arrow support
}

When it happens

Trigger: Adding an upstream column whose type maps to an unsupported Arrow DataType (e.g., large variants, dictionary/nested types, unsigned integers, nanosecond timestamps) that iceberg-rust's arrow_type_to_type rejects.

Common situations: `ALTER TABLE upstream ADD COLUMN x <exotic type>` propagating to an Iceberg sink; JSON/struct/list columns with unsupported nesting; timezone-less or precision-mismatched timestamp types.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• failed to convert : to arrow type
• convert parquet batch of
• Current iceberg schema does not match either original_schema
• Delete field id not found in schema
• error converting Arrow schema to Iceberg schema

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/b94b386db7eaac94. Report an issue: GitHub.

Appendix: source

Thrown at src/connector/src/sink/iceberg/commit.rs:952

        match schema_change.op.as_ref() {
            Some(risingwave_pb::stream_plan::sink_schema_change::Op::AddColumns(
                add_columns_op,
            )) => {
                let add_columns = add_columns_op.fields.iter().map(Field::from).collect_vec();
                for field in &add_columns {
                    // Convert RisingWave Field to Arrow Field using IcebergCreateTableArrowConvert
                    let arrow_field = iceberg_create_table_arrow_convert
                        .to_arrow_field(&field.name, &field.data_type)
                        .with_context(|| {
                            format!("Failed to convert field '{}' to arrow", field.name)
                        })
                        .map_err(SinkError::Iceberg)?;

                    // Convert Arrow DataType to Iceberg Type
                    let iceberg_type = iceberg::arrow::arrow_type_to_type(arrow_field.data_type())
                        .map_err(|err| {
                            SinkError::Iceberg(
                                anyhow!(err)
                                    .context("Failed to convert Arrow type to Iceberg type"),
                            )
                        })?;

                    new_fields.push(AddColumn::optional(&field.name, iceberg_type));
                    tracing::info!("Prepared field '{}' for schema change", field.name);
                }
            }
            Some(risingwave_pb::stream_plan::sink_schema_change::Op::DropColumns(
                drop_columns_op,
            )) => {
                drop_column_names = drop_columns_op.column_names.clone();
            }
            _ => {
                return Err(SinkError::Iceberg(anyhow!(
                    "Unsupported sink schema change op in iceberg sink: {:?}",
                    schema_change.op
                )));

View on GitHub (pinned to 6469eb736d)