{"record":{"id":"584b7584975b8a05","repo":"apache/beam","slug":"nullable-array-type-is-not-supported-in-datacatalog-schemas","errorCode":null,"errorMessage":"Nullable array type is not supported in DataCatalog schemas: {fieldType}","messagePattern":"Nullable array type is not supported in DataCatalog schemas: (.+?)","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java","lineNumber":116,"sourceCode":"    throw new UnsupportedOperationException(\n        \"Field type '\" + dcFieldType + \"' is not supported (field '\" + column.getColumn() + \"')\");\n  }\n\n  /** Convert Beam schema to DataCatalog schema. */\n  static com.google.cloud.datacatalog.v1beta1.Schema toDataCatalog(Schema schema) {\n    com.google.cloud.datacatalog.v1beta1.Schema.Builder schemaBuilder =\n        com.google.cloud.datacatalog.v1beta1.Schema.newBuilder();\n    for (Schema.Field field : schema.getFields()) {\n      schemaBuilder.addColumns(fromBeamField(field));\n    }\n    return schemaBuilder.build();\n  }\n\n  private static ColumnSchema fromBeamField(Schema.Field field) {\n    Schema.FieldType fieldType = field.getType();\n    if (fieldType.getTypeName().equals(Schema.TypeName.ARRAY)) {\n      if (fieldType.getNullable()) {\n        throw new UnsupportedOperationException(\n            \"Nullable array type is not supported in DataCatalog schemas: \" + fieldType);\n      } else if (fieldType.getCollectionElementType().getNullable()) {\n        throw new UnsupportedOperationException(\n            \"Nullable array element type is not supported in DataCatalog schemas: \" + fieldType);\n      } else if (fieldType.getCollectionElementType().getTypeName().equals(Schema.TypeName.ARRAY)) {\n        throw new UnsupportedOperationException(\n            \"Array of arrays not supported in DataCatalog schemas: \" + fieldType);\n      }\n      ColumnSchema column =\n          fromBeamField(Field.of(field.getName(), fieldType.getCollectionElementType()));\n      if (!column.getMode().equals(\"REQUIRED\")) {\n        // We should have bailed out earlier for any cases that would result in mode being set.\n        throw new AssertionError(\n            \"ColumnSchema for collection element type has non-empty mode: \" + fieldType);\n      }\n      return column.toBuilder().setMode(\"REPEATED\").build();\n    } else { // struct or primitive type\n      ColumnSchema.Builder colBuilder =","sourceCodeStart":98,"sourceCodeEnd":134,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java#L98-L134","documentation":"fromBeamField converts a Beam ARRAY field to Data Catalog and found the array type itself marked nullable. Data Catalog represents arrays only as REPEATED mode columns and has no way to express a nullable array-of-non-nullables, so the conversion throws; the nullable ARRAY field in the Beam schema is at fault.","triggerScenarios":"Calling SchemaUtils.toDataCatalog(schema) (or column()/fromBeamField) where a field's FieldType is ARRAY and getNullable() is true.","commonSituations":"Beam pipelines with Schema.builder().addNullableArrayField(...) being persisted to Data Catalog; default-nullability helpers silently creating nullable arrays.","solutions":["Declare the array field as non-nullable (Field.of(name, FieldType.array(...)) without nullable)","Wrap the whole field as nullable at a different level if semantics allow, or drop nullability","Convert nullable arrays to REQUIRED arrays with NULLABLE elements if DataCatalog can express it"],"exampleFix":"// before\nFieldType.array(elementType).withNullable(true)\n// after\nFieldType.array(elementType) // non-nullable array","handlingStrategy":"validation","validationCode":"for (Schema.Field f : beamSchema.getFields()) {\n  if (f.getType().getTypeName() == Schema.TypeName.ARRAY && f.getType().getNullable()) {\n    throw new IllegalArgumentException(\"Nullable array not allowed: \" + f.getName());\n  }\n}","typeGuard":"static boolean isNonNullableArray(Schema.FieldType t) {\n  return t.getTypeName() == Schema.TypeName.ARRAY && !t.getNullable();\n}","tryCatchPattern":"try {\n  com.google.cloud.datacatalog.v1beta1.Schema dc = SchemaUtils.toDataCatalog(beamSchema);\n} catch (UnsupportedOperationException ex) {\n  LOG.error(\"Schema not representable in DataCatalog\", ex);\n}","preventionTips":["Declare arrays non-nullable in Beam schemas intended for DataCatalog export","Review auto-generated schemas (JSON/Avro inference) for nullable arrays","Add schema round-trip tests DC<->Beam to catch nullability mismatches"],"tags":["datacatalog","schema","nullable-array"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}