apache/beam · error · UnsupportedOperationException

Nullable array type is not supported in DataCatalog schemas

Error message

Nullable array type is not supported in DataCatalog schemas: {fieldType}

What it means

fromBeamField converts a Beam ARRAY field to Data Catalog and found the array type itself marked nullable. Data Catalog represents arrays only as REPEATED mode columns and has no way to express a nullable array-of-non-nullables, so the conversion throws; the nullable ARRAY field in the Beam schema is at fault.

Solutions

  1. Declare the array field as non-nullable (Field.of(name, FieldType.array(...)) without nullable)
  2. Wrap the whole field as nullable at a different level if semantics allow, or drop nullability
  3. Convert nullable arrays to REQUIRED arrays with NULLABLE elements if DataCatalog can express it

Example fix

// before
FieldType.array(elementType).withNullable(true)
// after
FieldType.array(elementType) // non-nullable array
Defensive patterns

Strategy: validation

Validate before calling

for (Schema.Field f : beamSchema.getFields()) {
  if (f.getType().getTypeName() == Schema.TypeName.ARRAY && f.getType().getNullable()) {
    throw new IllegalArgumentException("Nullable array not allowed: " + f.getName());
  }
}

Type guard

static boolean isNonNullableArray(Schema.FieldType t) {
  return t.getTypeName() == Schema.TypeName.ARRAY && !t.getNullable();
}

Try / catch

try {
  com.google.cloud.datacatalog.v1beta1.Schema dc = SchemaUtils.toDataCatalog(beamSchema);
} catch (UnsupportedOperationException ex) {
  LOG.error("Schema not representable in DataCatalog", ex);
}

Prevention

When it happens

Trigger: Calling SchemaUtils.toDataCatalog(schema) (or column()/fromBeamField) where a field's FieldType is ARRAY and getNullable() is true.

Common situations: Beam pipelines with Schema.builder().addNullableArrayField(...) being persisted to Data Catalog; default-nullability helpers silently creating nullable arrays.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/584b7584975b8a05. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java:116

    throw new UnsupportedOperationException(
        "Field type '" + dcFieldType + "' is not supported (field '" + column.getColumn() + "')");
  }

  /** Convert Beam schema to DataCatalog schema. */
  static com.google.cloud.datacatalog.v1beta1.Schema toDataCatalog(Schema schema) {
    com.google.cloud.datacatalog.v1beta1.Schema.Builder schemaBuilder =
        com.google.cloud.datacatalog.v1beta1.Schema.newBuilder();
    for (Schema.Field field : schema.getFields()) {
      schemaBuilder.addColumns(fromBeamField(field));
    }
    return schemaBuilder.build();
  }

  private static ColumnSchema fromBeamField(Schema.Field field) {
    Schema.FieldType fieldType = field.getType();
    if (fieldType.getTypeName().equals(Schema.TypeName.ARRAY)) {
      if (fieldType.getNullable()) {
        throw new UnsupportedOperationException(
            "Nullable array type is not supported in DataCatalog schemas: " + fieldType);
      } else if (fieldType.getCollectionElementType().getNullable()) {
        throw new UnsupportedOperationException(
            "Nullable array element type is not supported in DataCatalog schemas: " + fieldType);
      } else if (fieldType.getCollectionElementType().getTypeName().equals(Schema.TypeName.ARRAY)) {
        throw new UnsupportedOperationException(
            "Array of arrays not supported in DataCatalog schemas: " + fieldType);
      }
      ColumnSchema column =
          fromBeamField(Field.of(field.getName(), fieldType.getCollectionElementType()));
      if (!column.getMode().equals("REQUIRED")) {
        // We should have bailed out earlier for any cases that would result in mode being set.
        throw new AssertionError(
            "ColumnSchema for collection element type has non-empty mode: " + fieldType);
      }
      return column.toBuilder().setMode("REPEATED").build();
    } else { // struct or primitive type
      ColumnSchema.Builder colBuilder =

View on GitHub (pinned to 12126d8942)