apache/iceberg · error · UnsupportedOperationException

User-defined types are not supported

Error message

User-defined types are not supported

What it means

SparkTypeVisitor.visit detects a Spark UserDefinedType (UDT) and immediately throws UnsupportedOperationException, because Iceberg has no representation for arbitrary user-defined Spark types. The visitor supports struct, array, map, variant and known atomic types only; UDTs are explicitly rejected so the traversal cannot silently misrepresent data.

Solutions

  1. Convert the UDT column to a supported base type before conversion (UDTs carry a sqlType; udf.sqlType() can be extracted or the column cast with .cast())
  2. Drop the UDT column from the schema if it is not needed in Iceberg
  3. Explode the UDT into primitive columns (e.g. vector -> array<double>) prior to writing to Iceberg
  4. If the UDT is essential, write via a format that preserves it or store a serialized representation (e.g. binary/string)

Example fix

// before
Types.StructType type = SparkSchemaUtil.convert(df.schema()); // VectorUDT column -> UnsupportedOperationException
// after
Dataset<Row> fixed = df.withColumn("features", col("features").cast("array<double>"));
Types.StructType type = SparkSchemaUtil.convert(fixed.schema());
Defensive patterns

Strategy: type-guard

Validate before calling

static boolean hasUdt(StructType schema) {
  for (StructField f : schema.fields()) {
    if (f.dataType() instanceof UserDefinedType) return true;
    if (f.dataType() instanceof StructType && hasUdt((StructType) f.dataType())) return true;
    if (f.dataType() instanceof ArrayType && ((ArrayType) f.dataType()).elementType() instanceof UserDefinedType) return true;
  }
  return false;
}

Type guard

if (!(dataType instanceof UserDefinedType)) { visit(dataType); }

Try / catch

try {
  SparkTypeVisitor.visit(sparkType, visitor);
} catch (UnsupportedOperationException e) {
  if (e.getMessage().contains("User-defined types")) { /* drop or cast the UDT column */ }
  throw e;
}

Prevention

When it happens

Trigger: Invoking SparkTypeVisitor.visit (directly or via schema conversion utilities) on a Spark type tree that contains a UserDefinedType field, e.g. converting a Spark schema containing a UDT column (such as Spark's internal VectorUDT from ML) to an Iceberg type.

Common situations: Ingesting DataFrames produced by MLlib pipelines (features column is a Vector UDT), or tables containing registered UDTs from third-party libraries, when exporting schema/data to Iceberg.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/ab5f8804d1b48230. Report an issue: GitHub.

Appendix: source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkTypeVisitor.java:56

        fieldResults.add(visitor.field(field, visit(field.dataType(), visitor)));
      }

      return visitor.struct((StructType) type, fieldResults);

    } else if (type instanceof MapType) {
      return visitor.map(
          (MapType) type,
          visit(((MapType) type).keyType(), visitor),
          visit(((MapType) type).valueType(), visitor));

    } else if (type instanceof ArrayType) {
      return visitor.array((ArrayType) type, visit(((ArrayType) type).elementType(), visitor));

    } else if (type instanceof VariantType) {
      return visitor.variant((VariantType) type);

    } else if (type instanceof UserDefinedType) {
      throw new UnsupportedOperationException("User-defined types are not supported");

    } else {
      return visitor.atomic(type);
    }
  }

  public T variant(VariantType variant) {
    throw new UnsupportedOperationException("Not implemented for variant");
  }

  public T struct(StructType struct, List<T> fieldResults) {
    return null;
  }

  public T field(StructField field, T typeResult) {
    return null;
  }

View on GitHub (pinned to 86d9c8fc54)