risingwavelabs/risingwave · error · SinkError::Iceberg

error converting Arrow schema to Iceberg schema: {err}

Error message

error converting Arrow schema to Iceberg schema: {err}

What it means

RisingWave's Iceberg sink failed to convert the projected Arrow schema of the stream chunk into an Iceberg schema while building the equality-delete upsert writer. The underlying error comes from iceberg-rust's `arrow_schema_to_schema`, which rejects Arrow types or field metadata that have no Iceberg equivalent (e.g. unsupported logical types, mismatched field id metadata). It is wrapped in `SinkError::Iceberg` and aborts sink construction.

Source

Thrown at src/connector/src/sink/iceberg/writer.rs:583

                    Some(format!("pos-del-{}", unique_uuid_suffix)),
                    iceberg::spec::DataFileFormat::Parquet,
                ),
            );
            PositionDeleteWriterBuilderType::PositionDelete(PositionDeleteFileWriterBuilder::new(
                rolling_writer_builder,
            ))
        };
        let equality_delete_builder = {
            let eq_del_config = EqualityDeleteWriterConfig::new(
                unique_column_ids.clone(),
                table.metadata().current_schema().clone(),
            )
            .map_err(|err| SinkError::Iceberg(anyhow!(err)))?;
            let parquet_writer_builder = ParquetWriterBuilder::new(
                parquet_writer_properties,
                Arc::new(
                    arrow_schema_to_schema(eq_del_config.projected_arrow_schema_ref())
                        .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,
                ),
            );
            let rolling_writer_builder = RollingFileWriterBuilder::new(
                parquet_writer_builder,
                (config.target_file_size_mb() * 1024 * 1024) as usize,
                table.file_io().clone(),
                DefaultLocationGenerator::new(table.metadata())
                    .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,
                DefaultFileNameGenerator::new(
                    writer_param.actor_id.to_string(),
                    Some(format!("eq-del-{}", unique_uuid_suffix)),
                    iceberg::spec::DataFileFormat::Parquet,
                ),
            );

            EqualityDeleteFileWriterBuilder::new(rolling_writer_builder, eq_del_config)
        };
        let delta_builder = DeltaWriterBuilder::new(

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the `err` detail to identify the exact column/type that failed the Arrow->Iceberg mapping and remove or reshape that column in the sink definition.
2. Restrict the sink/equality-delete projection to columns with plain Iceberg-supported types (int/long/float/double/string/boolean/date/timestamp/decimal/binary).
3. Recreate the table with a simpler schema, or alter the MV/sink so unsupported types are cast before reaching the sink.
4. Check the iceberg-rust dependency version matches the one RisingWave was built against and rebuild.

Example fix

// before: projecting a schema containing an unsupported Arrow type
let schema = arrow_schema_to_schema(eq_del_config.projected_arrow_schema_ref())?;
// after: cast unsupported columns to supported types before building the sink
let projected = projected_schema_without(&["weird_nested_col"]);
let schema = arrow_schema_to_schema(&projected)?;

// Ensure the projected Arrow schema only contains Iceberg-mappable types before building the sink
fn assert_arrow_types_iceberg_compatible(schema: &arrow_schema::Schema) -> Result<()> {
    for field in schema.fields() {
        match field.data_type() {
            arrow_schema::DataType::Null
            | arrow_schema::DataType::List(_)
            | arrow_schema::DataType::Dictionary(_, _)
            | arrow_schema::DataType::RunEndEncoded(_, _) => {
                anyhow::bail!("column {} has type not mappable to Iceberg", field.name())
            }
            _ => {}
        }
    }
    Ok(())
}

fn is_iceberg_mappable(dt: &arrow_schema::DataType) -> bool {
    !matches!(dt, arrow_schema::DataType::Null | arrow_schema::DataType::Dictionary(_, _) | arrow_schema::DataType::RunEndEncoded(_, _))
}

match arrow_schema_to_schema(eq_del_config.projected_arrow_schema_ref()) {
    Ok(schema) => schema,
    Err(e) => return Err(SinkError::Iceberg(anyhow!("arrow->iceberg conversion failed: {e:#}"))),
}

When it happens

Trigger: Calling `build_upsert` where `eq_del_config.projected_arrow_schema_ref()` contains Arrow field types that iceberg-rust cannot map to Iceberg types (e.g. certain nested or dictionary types), or Arrow fields missing/with malformed Iceberg field-id metadata produced by the earlier Iceberg->Arrow projection.

Common situations: Iceberg table schema with exotic column types (nested structs/maps/lists, time/timestamp variants) being projected into the equality-delete config; version drift between the vendored iceberg-rust and the Arrow version where type mappings changed; corrupted or absent field-id metadata on the Arrow schema.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• failed to convert arrow schema to iceberg schema
• error converting Iceberg schema to Arrow schema: {err}
• error converting StreamChunk to Arrow RecordBatch: {err}
• RecordBatch::try_new failed: {err}
• support negative scale for arrow decimal

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/12e52c7b793221c6. Report an issue: GitHub.