{"record":{"id":"d0e559e88f749dbd","repo":"risingwavelabs/risingwave","slug":"error-converting-iceberg-schema-to-arrow-schema","errorCode":null,"errorMessage":"error converting Iceberg schema to Arrow schema: {err}","messagePattern":"error converting Iceberg schema to Arrow schema: (.+?)","errorType":"exception","errorClass":"SinkError::Iceberg","httpStatus":null,"severity":"error","filePath":"src/connector/src/sink/iceberg/writer.rs","lineNumber":424,"sourceCode":"            write_qps.clone(),\n            write_latency.clone(),\n        );\n        let writer_builder = TaskWriterBuilderWrapper::new(\n            monitored_builder,\n            fanout_enabled,\n            schema.clone(),\n            partition_spec.clone(),\n            true,\n        );\n        let inner_writer = Some(Box::new(\n            writer_builder\n                .build()\n                .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,\n        ) as Box<dyn IcebergWriter>);\n        Ok(Self {\n            arrow_schema: Arc::new(\n                schema_to_arrow_schema(table.metadata().current_schema())\n                    .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,\n            ),\n            metrics: IcebergWriterMetrics {\n                _write_qps: write_qps,\n                _write_latency: write_latency,\n                write_bytes,\n            },\n            writer: IcebergWriterDispatch::Append {\n                writer: inner_writer,\n                writer_builder,\n            },\n            table,\n            actor_id: actor_id.to_string(),\n            sink_id: sink_id.to_string(),\n            sink_name: sink_name.clone(),\n            table_name,\n            writer_mode: \"append_only\",\n            project_idx_vec: {\n                if let Some(extra_partition_col_idx) = extra_partition_col_idx {","sourceCodeStart":406,"sourceCodeEnd":442,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/sink/iceberg/writer.rs#L406-L442","documentation":"Thrown in the IcebergWriteResult/append-only writer constructor when schema_to_arrow_schema fails to convert the Iceberg table schema into an Arrow schema. The Iceberg and Arrow type systems are not perfectly overlapping, so some Iceberg field types (or malformed field metadata) cannot be represented, aborting writer construction.","triggerScenarios":"Constructing the writer for a table whose current_schema contains Arrow-unsupported types (e.g. certain nested/unknown types or types added by newer Iceberg spec versions than the pinned iceberg-rs supports).","commonSituations":"Sink targeting a table with exotic column types (deeply nested structs, unknown types); iceberg-rs crate older than the table's spec features; schema evolved after the sink was created to add an unsupported column type.","solutions":["Inspect the table schema for column types unsupported by the conversion and alter them to supported types (e.g. primitives, standard lists/structs)","Upgrade the iceberg-rs / arrow crates so newer Iceberg types are convertible","Exclude unsupported columns from the sink's column list if business logic allows","Check the wrapped error to identify the exact field that failed conversion"],"exampleFix":"// before\nschema_to_arrow_schema(table.metadata().current_schema())\n    .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,\n// after\nschema_to_arrow_schema(table.metadata().current_schema()).map_err(|err| {\n    SinkError::Iceberg(anyhow::anyhow!(\n        \"error converting Iceberg schema to Arrow schema: {err}; fields: {:?}\",\n        table.metadata().current_schema().fields().map(|f| (f.id(), f.field_type.to_string())).collect::<Vec<_>>()\n    ))\n})?","handlingStrategy":"validation","validationCode":"// Ensure every Iceberg field type is convertible before constructing the writer\nfor f in table.metadata().current_schema().fields() {\n    if !matches!(f.field_type.as_primitive_type(), Some(_)) && !matches!(f.field_type, iceberg::spec::Type::Struct(_) | iceberg::spec::Type::List(_) | iceberg::spec::Type::Map(_)) {\n        return Err(format!(\"unsupported field type for Arrow: {}\", f.field_type));\n    }\n}","typeGuard":null,"tryCatchPattern":"schema_to_arrow_schema(table.metadata().current_schema())\n    .map_err(|err| SinkError::Iceberg(anyhow!(\"error converting Iceberg schema to Arrow schema: {err}\")))?","preventionTips":["Restrict sink tables to Arrow-representable Iceberg types","Keep iceberg-rs/arrow updated so newer Iceberg types convert cleanly","Re-test schema conversion after any table schema evolution","Log the offending field from the wrapped error"],"tags":["rust","iceberg","arrow","schema-conversion"],"backgroundTag":"type-mismatch","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}