{"record":{"id":"7215194d25715355","repo":"risingwavelabs/risingwave","slug":"failed-to-convert-arrow-schema-to-iceberg-schema","errorCode":null,"errorMessage":"failed to convert arrow schema to iceberg schema","messagePattern":"failed to convert arrow schema to iceberg schema","errorType":"exception","errorClass":"SinkError::Iceberg","httpStatus":null,"severity":"error","filePath":"src/connector/src/sink/iceberg/create_table.rs","lineNumber":146,"sourceCode":"\n    let iceberg_create_table_arrow_convert = IcebergCreateTableArrowConvert::default();\n    // convert risingwave schema -> arrow schema -> iceberg schema\n    let arrow_fields = param\n        .columns\n        .iter()\n        .map(|column| {\n            Ok(iceberg_create_table_arrow_convert\n                .to_arrow_field(&column.name, &column.data_type)\n                .map_err(|e| SinkError::Iceberg(anyhow!(e)))\n                .context(format!(\n                    \"failed to convert {}: {} to arrow type\",\n                    column.name, column.data_type\n                ))?)\n        })\n        .collect::<Result<Vec<ArrowField>>>()?;\n    let arrow_schema = arrow_schema_iceberg::Schema::new(arrow_fields);\n    let iceberg_schema = iceberg::arrow::arrow_schema_to_schema(&arrow_schema)\n        .map_err(|e| SinkError::Iceberg(anyhow!(e)))\n        .context(\"failed to convert arrow schema to iceberg schema\")?;\n\n    let location = {\n        let mut names = namespace.clone().inner();\n        names.push(table_name.clone());\n        match &config.common.warehouse_path {\n            Some(warehouse_path) => {\n                let is_s3_tables = warehouse_path.starts_with(\"arn:aws:s3tables\");\n                // Lakehouse Iceberg REST catalog federation uses bq:// prefix for BigQuery-managed Iceberg tables.\n                let is_bq_catalog_federation = warehouse_path.starts_with(\"bq://\");\n                let url = Url::parse(warehouse_path);\n                if url.is_err() || is_s3_tables || is_bq_catalog_federation {\n                    // For rest catalog, the warehouse_path could be a warehouse name.\n                    // In this case, we should specify the location when creating a table.\n                    if config\n                        .common\n                        .is_rest_catalog()\n                        .map_err(|err| SinkError::Config(anyhow!(err)))?","sourceCodeStart":128,"sourceCodeEnd":164,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/sink/iceberg/create_table.rs#L128-L164","documentation":"After building the Arrow schema from the sink columns, the code converts it into an Iceberg schema using iceberg::arrow::arrow_schema_to_schema. If that conversion fails (Arrow types that have no Iceberg representation, or malformed fields), the error is wrapped and given this static context message.","triggerScenarios":"create_table_if_not_exists_impl calls arrow_schema_to_schema on the constructed arrow_schema and it returns Err — e.g. a field type like unsupported time/nested types produced by the previous conversion step.","commonSituations":"Unsupported nested or temporal Arrow types in the sink schema; a column converted to an Arrow type the iceberg-rust version cannot map back to Iceberg; mismatched iceberg-rust feature flags or versions.","solutions":["Check the chained underlying error for the specific Arrow field that failed.","Simplify or cast the offending column's type to a basic Iceberg-supported type (int, bigint, varchar, etc.).","Ensure the sink has no unsupported nested/struct/list field shapes, or reshape the MV.","Verify the bundled iceberg-rust crate version supports the types in use."],"exampleFix":"// before: column typed as an unconvertible nested type\nCREATE SINK s FROM mv WITH (connector='iceberg', ...);\n// after: flatten or cast the complex column in the MV\nCREATE MATERIALIZED VIEW mv_flat AS SELECT struct_col.a, struct_col.b FROM mv_src;","handlingStrategy":"validation","validationCode":"// Sanity check the Arrow schema converts to an Iceberg schema before creating\nif let Err(e) = iceberg::arrow::arrow_schema_to_schema(&arrow_schema) {\n    eprintln!(\"unconvertible arrow schema: {e}\");\n}","typeGuard":null,"tryCatchPattern":"match result {\n    Err(e) if e.to_string().contains(\"arrow schema to iceberg\") => {\n        // reshape/cast the offending column in the MV, then retry\n    }\n    other => other?,\n}","preventionTips":["Prefer simple Iceberg-native types in sink schemas.","Flatten deeply nested structs when possible.","Keep iceberg-rust and RisingWave versions in sync.","Validate schema conversion in CI for new MV shapes."],"tags":["rust","iceberg","arrow","schema-conversion"],"backgroundTag":"schema-validation-failed","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}