{"record":{"id":"5bfc5f44c2b2fa04","repo":"apache/beam","slug":"array-of-arrays-not-supported-in-datacatalog-schemas","errorCode":null,"errorMessage":"Array of arrays not supported in DataCatalog schemas: {fieldType}","messagePattern":"Array of arrays not supported in DataCatalog schemas: (.+?)","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java","lineNumber":122,"sourceCode":"    com.google.cloud.datacatalog.v1beta1.Schema.Builder schemaBuilder =\n        com.google.cloud.datacatalog.v1beta1.Schema.newBuilder();\n    for (Schema.Field field : schema.getFields()) {\n      schemaBuilder.addColumns(fromBeamField(field));\n    }\n    return schemaBuilder.build();\n  }\n\n  private static ColumnSchema fromBeamField(Schema.Field field) {\n    Schema.FieldType fieldType = field.getType();\n    if (fieldType.getTypeName().equals(Schema.TypeName.ARRAY)) {\n      if (fieldType.getNullable()) {\n        throw new UnsupportedOperationException(\n            \"Nullable array type is not supported in DataCatalog schemas: \" + fieldType);\n      } else if (fieldType.getCollectionElementType().getNullable()) {\n        throw new UnsupportedOperationException(\n            \"Nullable array element type is not supported in DataCatalog schemas: \" + fieldType);\n      } else if (fieldType.getCollectionElementType().getTypeName().equals(Schema.TypeName.ARRAY)) {\n        throw new UnsupportedOperationException(\n            \"Array of arrays not supported in DataCatalog schemas: \" + fieldType);\n      }\n      ColumnSchema column =\n          fromBeamField(Field.of(field.getName(), fieldType.getCollectionElementType()));\n      if (!column.getMode().equals(\"REQUIRED\")) {\n        // We should have bailed out earlier for any cases that would result in mode being set.\n        throw new AssertionError(\n            \"ColumnSchema for collection element type has non-empty mode: \" + fieldType);\n      }\n      return column.toBuilder().setMode(\"REPEATED\").build();\n    } else { // struct or primitive type\n      ColumnSchema.Builder colBuilder =\n          ColumnSchema.newBuilder().setType(getDataCatalogType(fieldType));\n\n      if (fieldType.getNullable()) {\n        colBuilder.setMode(\"NULLABLE\");\n      } else {\n        colBuilder.setMode(\"REQUIRED\");","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java#L104-L140","documentation":"fromBeamField found a Beam field whose element type is itself an array (array of arrays). Data Catalog columns support only one level of REPEATED nesting, so a doubly-nested array cannot be expressed and the conversion is refused; the nested ARRAY field type in the Beam schema is the faulty input.","triggerScenarios":"toDataCatalog conversion where fieldType.getCollectionElementType().getTypeName() == ARRAY for an ARRAY field.","commonSituations":"Schemas with nested lists (e.g. ARRAY<ARRAY<STRING>>) imported from formats like JSON or Protobuf repeated-in-repeated being exported to Data Catalog.","solutions":["Flatten the nested arrays into a single level or restructure as a ROW containing an array","Drop or transform nested array fields before persisting the schema to Data Catalog","Use STRUCT<ARRAY<...>> if a nested representation is truly needed"],"exampleFix":"// before\nFieldType.array(FieldType.array(FieldType.string()))\n// after\nFieldType.array(FieldType.row(Schema.of(Schema.Field.of(\"elem\", FieldType.string()))))","handlingStrategy":"validation","validationCode":"for (Schema.Field f : beamSchema.getFields()) {\n  Schema.FieldType t = f.getType();\n  if (t.getTypeName() == Schema.TypeName.ARRAY\n      && t.getCollectionElementType().getTypeName() == Schema.TypeName.ARRAY) {\n    throw new IllegalArgumentException(\"Nested arrays not allowed: \" + f.getName());\n  }\n}","typeGuard":"static boolean isFlatArray(Schema.FieldType t) {\n  return t.getTypeName() == Schema.TypeName.ARRAY\n      && t.getCollectionElementType().getTypeName() != Schema.TypeName.ARRAY;\n}","tryCatchPattern":"try {\n  SchemaUtils.toDataCatalog(beamSchema);\n} catch (UnsupportedOperationException ex) {\n  LOG.error(\"Flatten nested arrays before export\", ex);\n}","preventionTips":["Flatten nested arrays at data-modeling time","Use STRUCT wrappers instead of array-of-array","Validate exported schemas in CI against DataCatalog constraints"],"tags":["datacatalog","schema","nested-array"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}