apache/beam · error · UnsupportedOperationException

Nullable array element type is not supported in DataCatalog…

Error message

Nullable array element type is not supported in DataCatalog schemas: {fieldType}

What it means

SchemaUtils.fromBeamField throws this when a Beam array field's element type is itself nullable. DataCatalog ColumnSchema requires array elements to be REQUIRED, so nullable elements cannot be represented.

Solutions

Make the array element type non-nullable: FieldType.array(FieldType.string().withNullable(false))
Normalize the schema before conversion, coercing element nullability to REQUIRED
Keep nullable semantics in the Beam pipeline but skip persisting such fields to Data Catalog

Example fix

// before
FieldType.array(FieldType.string().withNullable(true))
// after
FieldType.array(FieldType.string()) // REQUIRED elements

Defensive patterns

Strategy: validation

Validate before calling

for (Schema.Field f : beamSchema.getFields()) {
  Schema.FieldType t = f.getType();
  if (t.getTypeName() == Schema.TypeName.ARRAY && t.getCollectionElementType().getNullable()) {
    throw new IllegalArgumentException("Nullable array elements not allowed: " + f.getName());
  }
}

Type guard

static boolean hasRequiredElements(Schema.FieldType t) {
  return t.getTypeName() == Schema.TypeName.ARRAY && !t.getCollectionElementType().getNullable();
}

Try / catch

try {
  SchemaUtils.toDataCatalog(beamSchema);
} catch (UnsupportedOperationException ex) {
  LOG.error("Array element must be REQUIRED", ex);
}

Prevention

Build arrays with non-nullable element types
Coerce element nullability when importing schemas from other formats
Test toDataCatalog on all production schemas

When it happens

Trigger: toDataCatalog conversion where fieldType.getCollectionElementType().getNullable() is true for an ARRAY field.

Common situations: Arrays built from nullable element types (common when inferring schemas from JSON/Avro) being written back to Data Catalog.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8c84bd4d5defb32d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java:119

  /** Convert Beam schema to DataCatalog schema. */
  static com.google.cloud.datacatalog.v1beta1.Schema toDataCatalog(Schema schema) {
    com.google.cloud.datacatalog.v1beta1.Schema.Builder schemaBuilder =
        com.google.cloud.datacatalog.v1beta1.Schema.newBuilder();
    for (Schema.Field field : schema.getFields()) {
      schemaBuilder.addColumns(fromBeamField(field));
    }
    return schemaBuilder.build();
  }

  private static ColumnSchema fromBeamField(Schema.Field field) {
    Schema.FieldType fieldType = field.getType();
    if (fieldType.getTypeName().equals(Schema.TypeName.ARRAY)) {
      if (fieldType.getNullable()) {
        throw new UnsupportedOperationException(
            "Nullable array type is not supported in DataCatalog schemas: " + fieldType);
      } else if (fieldType.getCollectionElementType().getNullable()) {
        throw new UnsupportedOperationException(
            "Nullable array element type is not supported in DataCatalog schemas: " + fieldType);
      } else if (fieldType.getCollectionElementType().getTypeName().equals(Schema.TypeName.ARRAY)) {
        throw new UnsupportedOperationException(
            "Array of arrays not supported in DataCatalog schemas: " + fieldType);
      }
      ColumnSchema column =
          fromBeamField(Field.of(field.getName(), fieldType.getCollectionElementType()));
      if (!column.getMode().equals("REQUIRED")) {
        // We should have bailed out earlier for any cases that would result in mode being set.
        throw new AssertionError(
            "ColumnSchema for collection element type has non-empty mode: " + fieldType);
      }
      return column.toBuilder().setMode("REPEATED").build();
    } else { // struct or primitive type
      ColumnSchema.Builder colBuilder =
          ColumnSchema.newBuilder().setType(getDataCatalogType(fieldType));

      if (fieldType.getNullable()) {

View on GitHub (pinned to 12126d8942)