apache/beam · error · IllegalArgumentException

Array of collection is not supported in BigQuery.

Error message

Array of collection is not supported in BigQuery.

What it means

toTableFieldSchema (Beam Schema -> BigQuery TableSchema) throws this IllegalArgumentException when a Beam field is an array/collection whose element type is itself a collection or a map. BigQuery has no ARRAY<ARRAY<...>> or ARRAY<MAP<...>> types, so the converter refuses instead of silently producing an invalid TableSchema. The check is: after unwrapping the outer collection, if the element type is still a collection or map type, conversion fails.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java:567

  }

  private static List<TableFieldSchema> toTableFieldSchema(Schema schema) {
    List<TableFieldSchema> fields = new ArrayList<>(schema.getFieldCount());
    for (Field schemaField : schema.getFields()) {
      FieldType type = schemaField.getType();

      TableFieldSchema field = new TableFieldSchema().setName(schemaField.getName());
      if (schemaField.getDescription() != null && !"".equals(schemaField.getDescription())) {
        field.setDescription(schemaField.getDescription());
      }

      if (!schemaField.getType().getNullable()) {
        field.setMode(Mode.REQUIRED.toString());
      }
      if (type.getTypeName().isCollectionType()) {
        type = Preconditions.checkArgumentNotNull(type.getCollectionElementType());
        if (type.getTypeName().isCollectionType() || type.getTypeName().isMapType()) {
          throw new IllegalArgumentException("Array of collection is not supported in BigQuery.");
        }
        field.setMode(Mode.REPEATED.toString());
      }
      if (TypeName.ROW == type.getTypeName()) {
        Schema subType = Preconditions.checkArgumentNotNull(type.getRowSchema());
        field.setFields(toTableFieldSchema(subType));
      }
      if (TypeName.MAP == type.getTypeName()) {
        FieldType mapKeyType = Preconditions.checkArgumentNotNull(type.getMapKeyType());
        FieldType mapValueType = Preconditions.checkArgumentNotNull(type.getMapValueType());
        Schema mapSchema =
            Schema.builder()
                .addField(BIGQUERY_MAP_KEY_FIELD_NAME, mapKeyType)
                .addField(BIGQUERY_MAP_VALUE_FIELD_NAME, mapValueType)
                .build();
        type = FieldType.row(mapSchema);
        field.setFields(toTableFieldSchema(mapSchema));
        field.setMode(Mode.REPEATED.toString());

View on GitHub (pinned to 12126d8942)

Solutions

Flatten the schema: replace nested arrays with a single array of ROWs (STRUCT) if BigQuery can represent it, e.g. array of array -> array of struct.
For array-of-map, model it as a REPEATED RECORD with key/value fields (the same representation BigQueryUtils uses for top-level maps).
Restructure the Beam Schema before writing: convert nested collections to JSON/BYTES STRING columns or explode rows so no nested collection remains.

Example fix

// before
FieldType.bad = FieldType.array(FieldType.array(FieldType.INT64));
// after
Schema inner = Schema.builder().addField("v", FieldType.INT64).build();
FieldType.good = FieldType.array(FieldType.row(inner));

Defensive patterns

Strategy: validation

Validate before calling

static void assertNoNestedCollections(Schema schema) {
  for (Field f : schema.getFields()) {
    FieldType t = f.getType();
    if (t.getTypeName().isCollectionType()) {
      FieldType el = t.getCollectionElementType();
      if (el != null && (el.getTypeName().isCollectionType() || el.getTypeName().isMapType())) {
        throw new IllegalArgumentException("Field " + f.getName() + " is an array of collection/map; BigQuery sink rejects it");
      }
    }
  }
}

Type guard

static boolean isBqWritableCollection(FieldType t) {
  return !t.getTypeName().isCollectionType()
      || !java.util.Optional.ofNullable(t.getCollectionElementType())
          .map(e -> e.getTypeName().isCollectionType() || e.getTypeName().isMapType())
          .orElse(false);
}

Try / catch

try {
  bigQueryIO.write().withSchema(Schema.class.cast(beamSchema)) /* ... */;
} catch (IllegalArgumentException e) {
  if (String.valueOf(e.getMessage()).contains("Array of collection")) {
    throw new IllegalArgumentException("Flatten nested arrays/maps into arrays of ROW before writing to BigQuery", e);
  }
  throw e;
}

Prevention

Design Beam Schemas for BigQuery sinks with at most one nesting level of collections (array or map, never both/nested).
Use array-of-STRUCT for nested repeated data; BigQuery natively supports REPEATED RECORD.
Run schema linting on the Beam Schema before constructing BigQueryIO writes in tests.

When it happens

Trigger: Calling BigQueryUtils.toTableSchema / BigQueryIO.write with a Beam Schema containing nested arrays (e.g. FieldType.array(FieldType.array(...))) or arrays of maps (FieldType.array(FieldType.map(...))).

Common situations: PCollections whose rows carry nested list structures from JSON/Avro sources; building schemas programmatically with repeated nested fields; porting a Parquet/protobuf schema with repeated repeated fields to BigQuery sink.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4a5cba9854de568c. Report an issue: GitHub.