risingwavelabs/risingwave · error · SinkError::Iceberg

failed to convert arrow schema to iceberg schema

Error message

failed to convert arrow schema to iceberg schema

What it means

After building the Arrow schema from the sink columns, the code converts it into an Iceberg schema using iceberg::arrow::arrow_schema_to_schema. If that conversion fails (Arrow types that have no Iceberg representation, or malformed fields), the error is wrapped and given this static context message.

Source

Thrown at src/connector/src/sink/iceberg/create_table.rs:146

    let iceberg_create_table_arrow_convert = IcebergCreateTableArrowConvert::default();
    // convert risingwave schema -> arrow schema -> iceberg schema
    let arrow_fields = param
        .columns
        .iter()
        .map(|column| {
            Ok(iceberg_create_table_arrow_convert
                .to_arrow_field(&column.name, &column.data_type)
                .map_err(|e| SinkError::Iceberg(anyhow!(e)))
                .context(format!(
                    "failed to convert {}: {} to arrow type",
                    column.name, column.data_type
                ))?)
        })
        .collect::<Result<Vec<ArrowField>>>()?;
    let arrow_schema = arrow_schema_iceberg::Schema::new(arrow_fields);
    let iceberg_schema = iceberg::arrow::arrow_schema_to_schema(&arrow_schema)
        .map_err(|e| SinkError::Iceberg(anyhow!(e)))
        .context("failed to convert arrow schema to iceberg schema")?;

    let location = {
        let mut names = namespace.clone().inner();
        names.push(table_name.clone());
        match &config.common.warehouse_path {
            Some(warehouse_path) => {
                let is_s3_tables = warehouse_path.starts_with("arn:aws:s3tables");
                // Lakehouse Iceberg REST catalog federation uses bq:// prefix for BigQuery-managed Iceberg tables.
                let is_bq_catalog_federation = warehouse_path.starts_with("bq://");
                let url = Url::parse(warehouse_path);
                if url.is_err() || is_s3_tables || is_bq_catalog_federation {
                    // For rest catalog, the warehouse_path could be a warehouse name.
                    // In this case, we should specify the location when creating a table.
                    if config
                        .common
                        .is_rest_catalog()
                        .map_err(|err| SinkError::Config(anyhow!(err)))?

View on GitHub (pinned to 6469eb736d)

Solutions

1. Check the chained underlying error for the specific Arrow field that failed.
2. Simplify or cast the offending column's type to a basic Iceberg-supported type (int, bigint, varchar, etc.).
3. Ensure the sink has no unsupported nested/struct/list field shapes, or reshape the MV.
4. Verify the bundled iceberg-rust crate version supports the types in use.

Example fix

// before: column typed as an unconvertible nested type
CREATE SINK s FROM mv WITH (connector='iceberg', ...);
// after: flatten or cast the complex column in the MV
CREATE MATERIALIZED VIEW mv_flat AS SELECT struct_col.a, struct_col.b FROM mv_src;

// Sanity check the Arrow schema converts to an Iceberg schema before creating
if let Err(e) = iceberg::arrow::arrow_schema_to_schema(&arrow_schema) {
    eprintln!("unconvertible arrow schema: {e}");
}

match result {
    Err(e) if e.to_string().contains("arrow schema to iceberg") => {
        // reshape/cast the offending column in the MV, then retry
    }
    other => other?,
}

When it happens

Trigger: create_table_if_not_exists_impl calls arrow_schema_to_schema on the constructed arrow_schema and it returns Err — e.g. a field type like unsupported time/nested types produced by the previous conversion step.

Common situations: Unsupported nested or temporal Arrow types in the sink schema; a column converted to an Arrow type the iceberg-rust version cannot map back to Iceberg; mismatched iceberg-rust feature flags or versions.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• error converting Iceberg schema to Arrow schema: {err}
• failed to convert {}: {} to arrow type
• RecordBatch::try_new failed: {e}
• error converting Arrow schema to Iceberg schema: {err}
• support negative scale for arrow decimal

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/7215194d25715355. Report an issue: GitHub.