{"record":{"id":"8c84bd4d5defb32d","repo":"apache/beam","slug":"nullable-array-element-type-is-not-supported-in-datacatalog","errorCode":null,"errorMessage":"Nullable array element type is not supported in DataCatalog schemas: {fieldType}","messagePattern":"Nullable array element type is not supported in DataCatalog schemas: (.+?)","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java","lineNumber":119,"sourceCode":"\n  /** Convert Beam schema to DataCatalog schema. */\n  static com.google.cloud.datacatalog.v1beta1.Schema toDataCatalog(Schema schema) {\n    com.google.cloud.datacatalog.v1beta1.Schema.Builder schemaBuilder =\n        com.google.cloud.datacatalog.v1beta1.Schema.newBuilder();\n    for (Schema.Field field : schema.getFields()) {\n      schemaBuilder.addColumns(fromBeamField(field));\n    }\n    return schemaBuilder.build();\n  }\n\n  private static ColumnSchema fromBeamField(Schema.Field field) {\n    Schema.FieldType fieldType = field.getType();\n    if (fieldType.getTypeName().equals(Schema.TypeName.ARRAY)) {\n      if (fieldType.getNullable()) {\n        throw new UnsupportedOperationException(\n            \"Nullable array type is not supported in DataCatalog schemas: \" + fieldType);\n      } else if (fieldType.getCollectionElementType().getNullable()) {\n        throw new UnsupportedOperationException(\n            \"Nullable array element type is not supported in DataCatalog schemas: \" + fieldType);\n      } else if (fieldType.getCollectionElementType().getTypeName().equals(Schema.TypeName.ARRAY)) {\n        throw new UnsupportedOperationException(\n            \"Array of arrays not supported in DataCatalog schemas: \" + fieldType);\n      }\n      ColumnSchema column =\n          fromBeamField(Field.of(field.getName(), fieldType.getCollectionElementType()));\n      if (!column.getMode().equals(\"REQUIRED\")) {\n        // We should have bailed out earlier for any cases that would result in mode being set.\n        throw new AssertionError(\n            \"ColumnSchema for collection element type has non-empty mode: \" + fieldType);\n      }\n      return column.toBuilder().setMode(\"REPEATED\").build();\n    } else { // struct or primitive type\n      ColumnSchema.Builder colBuilder =\n          ColumnSchema.newBuilder().setType(getDataCatalogType(fieldType));\n\n      if (fieldType.getNullable()) {","sourceCodeStart":101,"sourceCodeEnd":137,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java#L101-L137","documentation":"SchemaUtils.fromBeamField throws this when a Beam array field's element type is itself nullable. DataCatalog ColumnSchema requires array elements to be REQUIRED, so nullable elements cannot be represented.","triggerScenarios":"toDataCatalog conversion where fieldType.getCollectionElementType().getNullable() is true for an ARRAY field.","commonSituations":"Arrays built from nullable element types (common when inferring schemas from JSON/Avro) being written back to Data Catalog.","solutions":["Make the array element type non-nullable: FieldType.array(FieldType.string().withNullable(false))","Normalize the schema before conversion, coercing element nullability to REQUIRED","Keep nullable semantics in the Beam pipeline but skip persisting such fields to Data Catalog"],"exampleFix":"// before\nFieldType.array(FieldType.string().withNullable(true))\n// after\nFieldType.array(FieldType.string()) // REQUIRED elements","handlingStrategy":"validation","validationCode":"for (Schema.Field f : beamSchema.getFields()) {\n  Schema.FieldType t = f.getType();\n  if (t.getTypeName() == Schema.TypeName.ARRAY && t.getCollectionElementType().getNullable()) {\n    throw new IllegalArgumentException(\"Nullable array elements not allowed: \" + f.getName());\n  }\n}","typeGuard":"static boolean hasRequiredElements(Schema.FieldType t) {\n  return t.getTypeName() == Schema.TypeName.ARRAY && !t.getCollectionElementType().getNullable();\n}","tryCatchPattern":"try {\n  SchemaUtils.toDataCatalog(beamSchema);\n} catch (UnsupportedOperationException ex) {\n  LOG.error(\"Array element must be REQUIRED\", ex);\n}","preventionTips":["Build arrays with non-nullable element types","Coerce element nullability when importing schemas from other formats","Test toDataCatalog on all production schemas"],"tags":["datacatalog","schema","nullable-array"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}