apache/beam · error · UnsupportedOperationException

Array of arrays not supported in DataCatalog schemas

Error message

Array of arrays not supported in DataCatalog schemas: {fieldType}

What it means

fromBeamField found a Beam field whose element type is itself an array (array of arrays). Data Catalog columns support only one level of REPEATED nesting, so a doubly-nested array cannot be expressed and the conversion is refused; the nested ARRAY field type in the Beam schema is the faulty input.

Solutions

Flatten the nested arrays into a single level or restructure as a ROW containing an array
Drop or transform nested array fields before persisting the schema to Data Catalog
Use STRUCT<ARRAY<...>> if a nested representation is truly needed

Example fix

// before
FieldType.array(FieldType.array(FieldType.string()))
// after
FieldType.array(FieldType.row(Schema.of(Schema.Field.of("elem", FieldType.string()))))

Defensive patterns

Strategy: validation

Validate before calling

for (Schema.Field f : beamSchema.getFields()) {
  Schema.FieldType t = f.getType();
  if (t.getTypeName() == Schema.TypeName.ARRAY
      && t.getCollectionElementType().getTypeName() == Schema.TypeName.ARRAY) {
    throw new IllegalArgumentException("Nested arrays not allowed: " + f.getName());
  }
}

Type guard

static boolean isFlatArray(Schema.FieldType t) {
  return t.getTypeName() == Schema.TypeName.ARRAY
      && t.getCollectionElementType().getTypeName() != Schema.TypeName.ARRAY;
}

Try / catch

try {
  SchemaUtils.toDataCatalog(beamSchema);
} catch (UnsupportedOperationException ex) {
  LOG.error("Flatten nested arrays before export", ex);
}

Prevention

Flatten nested arrays at data-modeling time
Use STRUCT wrappers instead of array-of-array
Validate exported schemas in CI against DataCatalog constraints

When it happens

Trigger: toDataCatalog conversion where fieldType.getCollectionElementType().getTypeName() == ARRAY for an ARRAY field.

Common situations: Schemas with nested lists (e.g. ARRAY<ARRAY<STRING>>) imported from formats like JSON or Protobuf repeated-in-repeated being exported to Data Catalog.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5bfc5f44c2b2fa04. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java:122

    com.google.cloud.datacatalog.v1beta1.Schema.Builder schemaBuilder =
        com.google.cloud.datacatalog.v1beta1.Schema.newBuilder();
    for (Schema.Field field : schema.getFields()) {
      schemaBuilder.addColumns(fromBeamField(field));
    }
    return schemaBuilder.build();
  }

  private static ColumnSchema fromBeamField(Schema.Field field) {
    Schema.FieldType fieldType = field.getType();
    if (fieldType.getTypeName().equals(Schema.TypeName.ARRAY)) {
      if (fieldType.getNullable()) {
        throw new UnsupportedOperationException(
            "Nullable array type is not supported in DataCatalog schemas: " + fieldType);
      } else if (fieldType.getCollectionElementType().getNullable()) {
        throw new UnsupportedOperationException(
            "Nullable array element type is not supported in DataCatalog schemas: " + fieldType);
      } else if (fieldType.getCollectionElementType().getTypeName().equals(Schema.TypeName.ARRAY)) {
        throw new UnsupportedOperationException(
            "Array of arrays not supported in DataCatalog schemas: " + fieldType);
      }
      ColumnSchema column =
          fromBeamField(Field.of(field.getName(), fieldType.getCollectionElementType()));
      if (!column.getMode().equals("REQUIRED")) {
        // We should have bailed out earlier for any cases that would result in mode being set.
        throw new AssertionError(
            "ColumnSchema for collection element type has non-empty mode: " + fieldType);
      }
      return column.toBuilder().setMode("REPEATED").build();
    } else { // struct or primitive type
      ColumnSchema.Builder colBuilder =
          ColumnSchema.newBuilder().setType(getDataCatalogType(fieldType));

      if (fieldType.getNullable()) {
        colBuilder.setMode("NULLABLE");
      } else {
        colBuilder.setMode("REQUIRED");

View on GitHub (pinned to 12126d8942)