{"record":{"id":"2c442a9c91e797a0","repo":"risingwavelabs/risingwave","slug":"failed-to-convert-to-arrow-type","errorCode":null,"errorMessage":"failed to convert {}: {} to arrow type","messagePattern":"failed to convert (.+?): (.+?) to arrow type","errorType":"exception","errorClass":"SinkError::Iceberg","httpStatus":null,"severity":"error","filePath":"src/connector/src/sink/iceberg/create_table.rs","lineNumber":137,"sourceCode":"            .columns\n            .iter()\n            .find(|column| column.data_type.contains_variant())\n    {\n        return Err(SinkError::Config(anyhow!(\n            \"creating an Iceberg table with VARIANT column `{}` requires `format_version = '3'`\",\n            column.name\n        )));\n    }\n\n    let iceberg_create_table_arrow_convert = IcebergCreateTableArrowConvert::default();\n    // convert risingwave schema -> arrow schema -> iceberg schema\n    let arrow_fields = param\n        .columns\n        .iter()\n        .map(|column| {\n            Ok(iceberg_create_table_arrow_convert\n                .to_arrow_field(&column.name, &column.data_type)\n                .map_err(|e| SinkError::Iceberg(anyhow!(e)))\n                .context(format!(\n                    \"failed to convert {}: {} to arrow type\",\n                    column.name, column.data_type\n                ))?)\n        })\n        .collect::<Result<Vec<ArrowField>>>()?;\n    let arrow_schema = arrow_schema_iceberg::Schema::new(arrow_fields);\n    let iceberg_schema = iceberg::arrow::arrow_schema_to_schema(&arrow_schema)\n        .map_err(|e| SinkError::Iceberg(anyhow!(e)))\n        .context(\"failed to convert arrow schema to iceberg schema\")?;\n\n    let location = {\n        let mut names = namespace.clone().inner();\n        names.push(table_name.clone());\n        match &config.common.warehouse_path {\n            Some(warehouse_path) => {\n                let is_s3_tables = warehouse_path.starts_with(\"arn:aws:s3tables\");\n                // Lakehouse Iceberg REST catalog federation uses bq:// prefix for BigQuery-managed Iceberg tables.","sourceCodeStart":119,"sourceCodeEnd":155,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/sink/iceberg/create_table.rs#L119-L155","documentation":"This contextual error reports that one of the sink's columns could not be converted from its RisingWave type to an Arrow field, via IcebergCreateTableArrowConvert::to_arrow_field. The message names the column and its RisingWave data type, with the underlying converter error chained as context.","triggerScenarios":"During create_table_if_not_exists_impl, mapping param.columns through to_arrow_field returns Err for a specific column — typically an unsupported RisingWave type (e.g. VARIANT under format v1/v2, struct/list with unsupported inner types) in the sink definition.","commonSituations":"Sinking columns whose types have no Iceberg/Arrow equivalent; VARIANT columns without format-version 3; deeply nested struct/list types with unsupported leaf types; type added in RisingWave newer than the iceberg-rust version in use.","solutions":["Read the chained underlying error to identify the offending type.","Cast the offending column to a supported type in the source query/mv (e.g. CAST(col AS VARCHAR)).","For VARIANT columns, set table.format-version='3'.","Remove unsupported columns from the sink definition.","Upgrade RisingWave/iceberg-rust if the type should be supported."],"exampleFix":"// before: sinking a jsonb column without v3\nCREATE MATERIALIZED VIEW mv AS SELECT payload FROM src; -- payload JSONB\n// after\nCREATE MATERIALIZED VIEW mv AS SELECT CAST(payload AS VARCHAR) AS payload FROM src;","handlingStrategy":"validation","validationCode":"// Verify all sink column types are convertible before creating the sink\nfor col in &param.columns {\n    if let Err(e) = convert.to_arrow_field(&col.name, &col.data_type) {\n        eprintln!(\"column {} ({}) unsupported: {e}\", col.name, col.data_type);\n    }\n}","typeGuard":null,"tryCatchPattern":"match result {\n    Err(e) if e.to_string().contains(\"failed to convert\") => {\n        // cast the named column to a supported type in the MV\n    }\n    other => other?,\n}","preventionTips":["Restrict sink columns to Iceberg-supported types (int, bigint, varchar, timestamp, struct, list, etc.).","Cast VARIANT/JSONB to VARCHAR when format-version < 3.","Avoid exotic nested types in materialized views feeding Iceberg sinks.","Test sink creation in a dev environment after adding new column types."],"tags":["rust","iceberg","arrow","type-conversion"],"backgroundTag":"type-mismatch","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}