{"record":{"id":"b94b386db7eaac94","repo":"risingwavelabs/risingwave","slug":"failed-to-convert-arrow-type-to-iceberg-type","errorCode":null,"errorMessage":"Failed to convert Arrow type to Iceberg type","messagePattern":"Failed to convert Arrow type to Iceberg type","errorType":"exception","errorClass":"SinkError::Iceberg","httpStatus":null,"severity":"error","filePath":"src/connector/src/sink/iceberg/commit.rs","lineNumber":952,"sourceCode":"        match schema_change.op.as_ref() {\n            Some(risingwave_pb::stream_plan::sink_schema_change::Op::AddColumns(\n                add_columns_op,\n            )) => {\n                let add_columns = add_columns_op.fields.iter().map(Field::from).collect_vec();\n                for field in &add_columns {\n                    // Convert RisingWave Field to Arrow Field using IcebergCreateTableArrowConvert\n                    let arrow_field = iceberg_create_table_arrow_convert\n                        .to_arrow_field(&field.name, &field.data_type)\n                        .with_context(|| {\n                            format!(\"Failed to convert field '{}' to arrow\", field.name)\n                        })\n                        .map_err(SinkError::Iceberg)?;\n\n                    // Convert Arrow DataType to Iceberg Type\n                    let iceberg_type = iceberg::arrow::arrow_type_to_type(arrow_field.data_type())\n                        .map_err(|err| {\n                            SinkError::Iceberg(\n                                anyhow!(err)\n                                    .context(\"Failed to convert Arrow type to Iceberg type\"),\n                            )\n                        })?;\n\n                    new_fields.push(AddColumn::optional(&field.name, iceberg_type));\n                    tracing::info!(\"Prepared field '{}' for schema change\", field.name);\n                }\n            }\n            Some(risingwave_pb::stream_plan::sink_schema_change::Op::DropColumns(\n                drop_columns_op,\n            )) => {\n                drop_column_names = drop_columns_op.column_names.clone();\n            }\n            _ => {\n                return Err(SinkError::Iceberg(anyhow!(\n                    \"Unsupported sink schema change op in iceberg sink: {:?}\",\n                    schema_change.op\n                )));","sourceCodeStart":934,"sourceCodeEnd":970,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/sink/iceberg/commit.rs#L934-L970","documentation":"commit_schema_change_impl converts each added column's Arrow DataType to an Iceberg Type via `iceberg::arrow::arrow_type_to_type`; some Arrow types have no Iceberg representation, and the conversion returns an error which is wrapped with this context. The schema evolution commit is aborted before any metadata update.","triggerScenarios":"Adding an upstream column whose type maps to an unsupported Arrow DataType (e.g., large variants, dictionary/nested types, unsigned integers, nanosecond timestamps) that iceberg-rust's arrow_type_to_type rejects.","commonSituations":"`ALTER TABLE upstream ADD COLUMN x <exotic type>` propagating to an Iceberg sink; JSON/struct/list columns with unsupported nesting; timezone-less or precision-mismatched timestamp types.","solutions":["Change the added column's type to an Iceberg-compatible one (e.g., timestamp, decimal, string, int/float/bool, supported nested types).","Cast the column to a supported type in the upstream/stream plan before it reaches the sink.","Drop and re-add the column with a supported type, then let schema evolution re-run.","Upgrade iceberg-rust/RisingWave if a newer version supports the Arrow type."],"exampleFix":"// before: adding unsupported type directly\nnew_fields.push(AddColumn::optional(&field.name, iceberg_type));\n// after: cast unsupported types in the upstream plan, e.g. to string\n// ALTER TABLE t ADD COLUMN flags jsonb  ->  cast in sink plan:\n// flags::varchar, then iceberg_type = arrow_type_to_type(Utf8)?","handlingStrategy":"validation","validationCode":"// Check the column type is Iceberg-representable before adding it upstream:\nfn is_iceberg_comparable(dt: &arrow::datatypes::DataType) -> bool {\n    use arrow::datatypes::DataType::*;\n    matches!(dt, Null | Boolean | Int8 | Int16 | Int32 | Int64 | Float32 | Float64 | Utf8 | Binary | Date32 | Timestamp(_, _) | Decimal128(_, _) | _) // tighten per iceberg::arrow support\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Use Iceberg-native column types (string, int, bigint, timestamp, decimal) for new upstream columns.","Cast unsupported types in the stream plan before the sink.","Test schema evolution on a staging table before production."],"tags":["iceberg","arrow","type-conversion","schema-evolution"],"backgroundTag":"type-mismatch","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}