{"record":{"id":"4a5cba9854de568c","repo":"apache/beam","slug":"array-of-collection-is-not-supported-in-bigquery","errorCode":null,"errorMessage":"Array of collection is not supported in BigQuery.","messagePattern":"Array of collection is not supported in BigQuery\\.","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java","lineNumber":567,"sourceCode":"  }\n\n  private static List<TableFieldSchema> toTableFieldSchema(Schema schema) {\n    List<TableFieldSchema> fields = new ArrayList<>(schema.getFieldCount());\n    for (Field schemaField : schema.getFields()) {\n      FieldType type = schemaField.getType();\n\n      TableFieldSchema field = new TableFieldSchema().setName(schemaField.getName());\n      if (schemaField.getDescription() != null && !\"\".equals(schemaField.getDescription())) {\n        field.setDescription(schemaField.getDescription());\n      }\n\n      if (!schemaField.getType().getNullable()) {\n        field.setMode(Mode.REQUIRED.toString());\n      }\n      if (type.getTypeName().isCollectionType()) {\n        type = Preconditions.checkArgumentNotNull(type.getCollectionElementType());\n        if (type.getTypeName().isCollectionType() || type.getTypeName().isMapType()) {\n          throw new IllegalArgumentException(\"Array of collection is not supported in BigQuery.\");\n        }\n        field.setMode(Mode.REPEATED.toString());\n      }\n      if (TypeName.ROW == type.getTypeName()) {\n        Schema subType = Preconditions.checkArgumentNotNull(type.getRowSchema());\n        field.setFields(toTableFieldSchema(subType));\n      }\n      if (TypeName.MAP == type.getTypeName()) {\n        FieldType mapKeyType = Preconditions.checkArgumentNotNull(type.getMapKeyType());\n        FieldType mapValueType = Preconditions.checkArgumentNotNull(type.getMapValueType());\n        Schema mapSchema =\n            Schema.builder()\n                .addField(BIGQUERY_MAP_KEY_FIELD_NAME, mapKeyType)\n                .addField(BIGQUERY_MAP_VALUE_FIELD_NAME, mapValueType)\n                .build();\n        type = FieldType.row(mapSchema);\n        field.setFields(toTableFieldSchema(mapSchema));\n        field.setMode(Mode.REPEATED.toString());","sourceCodeStart":549,"sourceCodeEnd":585,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java#L549-L585","documentation":"toTableFieldSchema (Beam Schema -> BigQuery TableSchema) throws this IllegalArgumentException when a Beam field is an array/collection whose element type is itself a collection or a map. BigQuery has no ARRAY<ARRAY<...>> or ARRAY<MAP<...>> types, so the converter refuses instead of silently producing an invalid TableSchema. The check is: after unwrapping the outer collection, if the element type is still a collection or map type, conversion fails.","triggerScenarios":"Calling BigQueryUtils.toTableSchema / BigQueryIO.write with a Beam Schema containing nested arrays (e.g. FieldType.array(FieldType.array(...))) or arrays of maps (FieldType.array(FieldType.map(...))).","commonSituations":"PCollections whose rows carry nested list structures from JSON/Avro sources; building schemas programmatically with repeated nested fields; porting a Parquet/protobuf schema with repeated repeated fields to BigQuery sink.","solutions":["Flatten the schema: replace nested arrays with a single array of ROWs (STRUCT) if BigQuery can represent it, e.g. array of array -> array of struct.","For array-of-map, model it as a REPEATED RECORD with key/value fields (the same representation BigQueryUtils uses for top-level maps).","Restructure the Beam Schema before writing: convert nested collections to JSON/BYTES STRING columns or explode rows so no nested collection remains."],"exampleFix":"// before\nFieldType.bad = FieldType.array(FieldType.array(FieldType.INT64));\n// after\nSchema inner = Schema.builder().addField(\"v\", FieldType.INT64).build();\nFieldType.good = FieldType.array(FieldType.row(inner));","handlingStrategy":"validation","validationCode":"static void assertNoNestedCollections(Schema schema) {\n  for (Field f : schema.getFields()) {\n    FieldType t = f.getType();\n    if (t.getTypeName().isCollectionType()) {\n      FieldType el = t.getCollectionElementType();\n      if (el != null && (el.getTypeName().isCollectionType() || el.getTypeName().isMapType())) {\n        throw new IllegalArgumentException(\"Field \" + f.getName() + \" is an array of collection/map; BigQuery sink rejects it\");\n      }\n    }\n  }\n}","typeGuard":"static boolean isBqWritableCollection(FieldType t) {\n  return !t.getTypeName().isCollectionType()\n      || !java.util.Optional.ofNullable(t.getCollectionElementType())\n          .map(e -> e.getTypeName().isCollectionType() || e.getTypeName().isMapType())\n          .orElse(false);\n}","tryCatchPattern":"try {\n  bigQueryIO.write().withSchema(Schema.class.cast(beamSchema)) /* ... */;\n} catch (IllegalArgumentException e) {\n  if (String.valueOf(e.getMessage()).contains(\"Array of collection\")) {\n    throw new IllegalArgumentException(\"Flatten nested arrays/maps into arrays of ROW before writing to BigQuery\", e);\n  }\n  throw e;\n}","preventionTips":["Design Beam Schemas for BigQuery sinks with at most one nesting level of collections (array or map, never both/nested).","Use array-of-STRUCT for nested repeated data; BigQuery natively supports REPEATED RECORD.","Run schema linting on the Beam Schema before constructing BigQueryIO writes in tests."],"tags":["bigquery","beam","schema-conversion","nested-array","write"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}