{"record":{"id":"12e52c7b793221c6","repo":"risingwavelabs/risingwave","slug":"error-converting-arrow-schema-to-iceberg-schema","errorCode":null,"errorMessage":"error converting Arrow schema to Iceberg schema: {err}","messagePattern":"error converting Arrow schema to Iceberg schema: (.+?)","errorType":"exception","errorClass":"SinkError::Iceberg","httpStatus":null,"severity":"error","filePath":"src/connector/src/sink/iceberg/writer.rs","lineNumber":583,"sourceCode":"                    Some(format!(\"pos-del-{}\", unique_uuid_suffix)),\n                    iceberg::spec::DataFileFormat::Parquet,\n                ),\n            );\n            PositionDeleteWriterBuilderType::PositionDelete(PositionDeleteFileWriterBuilder::new(\n                rolling_writer_builder,\n            ))\n        };\n        let equality_delete_builder = {\n            let eq_del_config = EqualityDeleteWriterConfig::new(\n                unique_column_ids.clone(),\n                table.metadata().current_schema().clone(),\n            )\n            .map_err(|err| SinkError::Iceberg(anyhow!(err)))?;\n            let parquet_writer_builder = ParquetWriterBuilder::new(\n                parquet_writer_properties,\n                Arc::new(\n                    arrow_schema_to_schema(eq_del_config.projected_arrow_schema_ref())\n                        .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,\n                ),\n            );\n            let rolling_writer_builder = RollingFileWriterBuilder::new(\n                parquet_writer_builder,\n                (config.target_file_size_mb() * 1024 * 1024) as usize,\n                table.file_io().clone(),\n                DefaultLocationGenerator::new(table.metadata())\n                    .map_err(|err| SinkError::Iceberg(anyhow!(err)))?,\n                DefaultFileNameGenerator::new(\n                    writer_param.actor_id.to_string(),\n                    Some(format!(\"eq-del-{}\", unique_uuid_suffix)),\n                    iceberg::spec::DataFileFormat::Parquet,\n                ),\n            );\n\n            EqualityDeleteFileWriterBuilder::new(rolling_writer_builder, eq_del_config)\n        };\n        let delta_builder = DeltaWriterBuilder::new(","sourceCodeStart":565,"sourceCodeEnd":601,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/sink/iceberg/writer.rs#L565-L601","documentation":"RisingWave's Iceberg sink failed to convert the projected Arrow schema of the stream chunk into an Iceberg schema while building the equality-delete upsert writer. The underlying error comes from iceberg-rust's `arrow_schema_to_schema`, which rejects Arrow types or field metadata that have no Iceberg equivalent (e.g. unsupported logical types, mismatched field id metadata). It is wrapped in `SinkError::Iceberg` and aborts sink construction.","triggerScenarios":"Calling `build_upsert` where `eq_del_config.projected_arrow_schema_ref()` contains Arrow field types that iceberg-rust cannot map to Iceberg types (e.g. certain nested or dictionary types), or Arrow fields missing/with malformed Iceberg field-id metadata produced by the earlier Iceberg->Arrow projection.","commonSituations":"Iceberg table schema with exotic column types (nested structs/maps/lists, time/timestamp variants) being projected into the equality-delete config; version drift between the vendored iceberg-rust and the Arrow version where type mappings changed; corrupted or absent field-id metadata on the Arrow schema.","solutions":["Inspect the `err` detail to identify the exact column/type that failed the Arrow->Iceberg mapping and remove or reshape that column in the sink definition.","Restrict the sink/equality-delete projection to columns with plain Iceberg-supported types (int/long/float/double/string/boolean/date/timestamp/decimal/binary).","Recreate the table with a simpler schema, or alter the MV/sink so unsupported types are cast before reaching the sink.","Check the iceberg-rust dependency version matches the one RisingWave was built against and rebuild."],"exampleFix":"// before: projecting a schema containing an unsupported Arrow type\nlet schema = arrow_schema_to_schema(eq_del_config.projected_arrow_schema_ref())?;\n// after: cast unsupported columns to supported types before building the sink\nlet projected = projected_schema_without(&[\"weird_nested_col\"]);\nlet schema = arrow_schema_to_schema(&projected)?;","handlingStrategy":"validation","validationCode":"// Ensure the projected Arrow schema only contains Iceberg-mappable types before building the sink\nfn assert_arrow_types_iceberg_compatible(schema: &arrow_schema::Schema) -> Result<()> {\n    for field in schema.fields() {\n        match field.data_type() {\n            arrow_schema::DataType::Null\n            | arrow_schema::DataType::List(_)\n            | arrow_schema::DataType::Dictionary(_, _)\n            | arrow_schema::DataType::RunEndEncoded(_, _) => {\n                anyhow::bail!(\"column {} has type not mappable to Iceberg\", field.name())\n            }\n            _ => {}\n        }\n    }\n    Ok(())\n}","typeGuard":"fn is_iceberg_mappable(dt: &arrow_schema::DataType) -> bool {\n    !matches!(dt, arrow_schema::DataType::Null | arrow_schema::DataType::Dictionary(_, _) | arrow_schema::DataType::RunEndEncoded(_, _))\n}","tryCatchPattern":"match arrow_schema_to_schema(eq_del_config.projected_arrow_schema_ref()) {\n    Ok(schema) => schema,\n    Err(e) => return Err(SinkError::Iceberg(anyhow!(\"arrow->iceberg conversion failed: {e:#}\"))),\n}","preventionTips":["Keep sink and MV columns restricted to Iceberg-supported primitive types","Never alter MV schemas while an Iceberg sink is attached; recreate the sink after changes","Pin the iceberg-rust dependency to the workspace version when upgrading"],"tags":["iceberg","arrow","schema-conversion","sink"],"backgroundTag":"schema-validation-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}