risingwavelabs/risingwave · error · SinkError::Iceberg

error converting Iceberg schema to Arrow schema: {err}

Error message

error converting Iceberg schema to Arrow schema: {err}

What it means

Thrown in the IcebergWriteResult/append-only writer constructor when schema_to_arrow_schema fails to convert the Iceberg table schema into an Arrow schema. The Iceberg and Arrow type systems are not perfectly overlapping, so some Iceberg field types (or malformed field metadata) cannot be represented, aborting writer construction.

Source

Thrown at src/connector/src/sink/iceberg/writer.rs:424

            write_qps.clone(),
            write_latency.clone(),
        );
        let writer_builder = TaskWriterBuilderWrapper::new(
            monitored_builder,
            fanout_enabled,
            schema.clone(),
            partition_spec.clone(),
            true,
        );
        let inner_writer = Some(Box::new(
            writer_builder
                .build()
                .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,
        ) as Box<dyn IcebergWriter>);
        Ok(Self {
            arrow_schema: Arc::new(
                schema_to_arrow_schema(table.metadata().current_schema())
                    .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,
            ),
            metrics: IcebergWriterMetrics {
                _write_qps: write_qps,
                _write_latency: write_latency,
                write_bytes,
            },
            writer: IcebergWriterDispatch::Append {
                writer: inner_writer,
                writer_builder,
            },
            table,
            actor_id: actor_id.to_string(),
            sink_id: sink_id.to_string(),
            sink_name: sink_name.clone(),
            table_name,
            writer_mode: "append_only",
            project_idx_vec: {
                if let Some(extra_partition_col_idx) = extra_partition_col_idx {

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the table schema for column types unsupported by the conversion and alter them to supported types (e.g. primitives, standard lists/structs)
2. Upgrade the iceberg-rs / arrow crates so newer Iceberg types are convertible
3. Exclude unsupported columns from the sink's column list if business logic allows
4. Check the wrapped error to identify the exact field that failed conversion

Example fix

// before
schema_to_arrow_schema(table.metadata().current_schema())
    .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,
// after
schema_to_arrow_schema(table.metadata().current_schema()).map_err(|err| {
    SinkError::Iceberg(anyhow::anyhow!(
        "error converting Iceberg schema to Arrow schema: {err}; fields: {:?}",
        table.metadata().current_schema().fields().map(|f| (f.id(), f.field_type.to_string())).collect::<Vec<_>>()
    ))
})?

// Ensure every Iceberg field type is convertible before constructing the writer
for f in table.metadata().current_schema().fields() {
    if !matches!(f.field_type.as_primitive_type(), Some(_)) && !matches!(f.field_type, iceberg::spec::Type::Struct(_) | iceberg::spec::Type::List(_) | iceberg::spec::Type::Map(_)) {
        return Err(format!("unsupported field type for Arrow: {}", f.field_type));
    }
}

schema_to_arrow_schema(table.metadata().current_schema())
    .map_err(|err| SinkError::Iceberg(anyhow!("error converting Iceberg schema to Arrow schema: {err}")))?

When it happens

Trigger: Constructing the writer for a table whose current_schema contains Arrow-unsupported types (e.g. certain nested/unknown types or types added by newer Iceberg spec versions than the pinned iceberg-rs supports).

Common situations: Sink targeting a table with exotic column types (deeply nested structs, unknown types); iceberg-rs crate older than the table's spec features; schema evolved after the sink was created to add an unsupported column type.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• failed to convert arrow schema to iceberg schema
• failed to convert {}: {} to arrow type
• RecordBatch::try_new failed: {e}
• error converting Arrow schema to Iceberg schema: {err}
• support negative scale for arrow decimal

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/d0e559e88f749dbd. Report an issue: GitHub.