{"record":{"id":"ab5f8804d1b48230","repo":"apache/iceberg","slug":"user-defined-types-are-not-supported-ab5f88","errorCode":null,"errorMessage":"User-defined types are not supported","messagePattern":"User-defined types are not supported","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkTypeVisitor.java","lineNumber":56,"sourceCode":"        fieldResults.add(visitor.field(field, visit(field.dataType(), visitor)));\n      }\n\n      return visitor.struct((StructType) type, fieldResults);\n\n    } else if (type instanceof MapType) {\n      return visitor.map(\n          (MapType) type,\n          visit(((MapType) type).keyType(), visitor),\n          visit(((MapType) type).valueType(), visitor));\n\n    } else if (type instanceof ArrayType) {\n      return visitor.array((ArrayType) type, visit(((ArrayType) type).elementType(), visitor));\n\n    } else if (type instanceof VariantType) {\n      return visitor.variant((VariantType) type);\n\n    } else if (type instanceof UserDefinedType) {\n      throw new UnsupportedOperationException(\"User-defined types are not supported\");\n\n    } else {\n      return visitor.atomic(type);\n    }\n  }\n\n  public T variant(VariantType variant) {\n    throw new UnsupportedOperationException(\"Not implemented for variant\");\n  }\n\n  public T struct(StructType struct, List<T> fieldResults) {\n    return null;\n  }\n\n  public T field(StructField field, T typeResult) {\n    return null;\n  }\n","sourceCodeStart":38,"sourceCodeEnd":74,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkTypeVisitor.java#L38-L74","documentation":"SparkTypeVisitor.visit detects a Spark UserDefinedType (UDT) and immediately throws UnsupportedOperationException, because Iceberg has no representation for arbitrary user-defined Spark types. The visitor supports struct, array, map, variant and known atomic types only; UDTs are explicitly rejected so the traversal cannot silently misrepresent data.","triggerScenarios":"Invoking SparkTypeVisitor.visit (directly or via schema conversion utilities) on a Spark type tree that contains a UserDefinedType field, e.g. converting a Spark schema containing a UDT column (such as Spark's internal VectorUDT from ML) to an Iceberg type.","commonSituations":"Ingesting DataFrames produced by MLlib pipelines (features column is a Vector UDT), or tables containing registered UDTs from third-party libraries, when exporting schema/data to Iceberg.","solutions":["Convert the UDT column to a supported base type before conversion (UDTs carry a sqlType; udf.sqlType() can be extracted or the column cast with .cast())","Drop the UDT column from the schema if it is not needed in Iceberg","Explode the UDT into primitive columns (e.g. vector -> array<double>) prior to writing to Iceberg","If the UDT is essential, write via a format that preserves it or store a serialized representation (e.g. binary/string)"],"exampleFix":"// before\nTypes.StructType type = SparkSchemaUtil.convert(df.schema()); // VectorUDT column -> UnsupportedOperationException\n// after\nDataset<Row> fixed = df.withColumn(\"features\", col(\"features\").cast(\"array<double>\"));\nTypes.StructType type = SparkSchemaUtil.convert(fixed.schema());","handlingStrategy":"type-guard","validationCode":"static boolean hasUdt(StructType schema) {\n  for (StructField f : schema.fields()) {\n    if (f.dataType() instanceof UserDefinedType) return true;\n    if (f.dataType() instanceof StructType && hasUdt((StructType) f.dataType())) return true;\n    if (f.dataType() instanceof ArrayType && ((ArrayType) f.dataType()).elementType() instanceof UserDefinedType) return true;\n  }\n  return false;\n}","typeGuard":"if (!(dataType instanceof UserDefinedType)) { visit(dataType); }","tryCatchPattern":"try {\n  SparkTypeVisitor.visit(sparkType, visitor);\n} catch (UnsupportedOperationException e) {\n  if (e.getMessage().contains(\"User-defined types\")) { /* drop or cast the UDT column */ }\n  throw e;\n}","preventionTips":["Audit schemas for UDT columns (common in MLlib outputs) before Iceberg conversion","Cast UDT columns to their underlying sqlType early in the pipeline","Do not persist ML Vector/UDT columns directly into Iceberg tables"],"tags":["unsupported-operation","spark","user-defined-type"],"backgroundTag":"unsupported-operation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}