apache/beam · error · IllegalArgumentException

Cannot convert BigQuery type '' to '' because the BigQuery t

Error message

Cannot convert BigQuery type '' to '' because the BigQuery type is a List, while the output type is not a collection.

What it means

Thrown in BigQueryUtils.fromJsonToBeamField when the BigQuery JSON value is a List (a BigQuery ARRAY/REPEATED field) but the target Beam FieldType has no collection element type, i.e. the Beam schema says the field is not an array. The message interpolates the value's Java class and the FieldType. It signals a schema mismatch between the BigQuery source and the declared Beam Row schema.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java:959

      } else if (fieldType.isLogicalType(Timestamp.IDENTIFIER)) {
        if (!jsonBQString.contains("UTC")) {
          BigDecimal bd = new BigDecimal(jsonBQString);
          long seconds = bd.longValue();
          long nanos = bd.subtract(BigDecimal.valueOf(seconds)).movePointRight(9).longValue();
          return java.time.Instant.ofEpochSecond(seconds, nanos);
        }
        return VAR_PRECISION_FORMATTER.parse(jsonBQString, java.time.Instant::from);
      }
    }

    if (jsonBQValue instanceof byte[] && fieldType.getTypeName() == TypeName.BYTES) {
      return jsonBQValue;
    }

    if (jsonBQValue instanceof List) {
      FieldType collectionElementType = fieldType.getCollectionElementType();
      if (collectionElementType == null) {
        throw new IllegalArgumentException(
            "Cannot convert BigQuery type '"
                + jsonBQValue.getClass()
                + "' to '"
                + fieldType
                + "' because the BigQuery type is a List, while the output type is not a"
                + " collection.");
      }

      boolean innerTypeIsMap = collectionElementType.getTypeName().isMapType();

      return ((List<@Nullable Object>) jsonBQValue)
          .stream()
              // Old BigQuery client returns arrays as lists of maps {"v": <value>}.
              // If this is the case, unwrap the value first
              .<@Nullable Object>map(
                  v ->
                      (!innerTypeIsMap
                              && v instanceof Map

View on GitHub (pinned to 12126d8942)

Solutions

Declare the field as an array in the Beam schema: FieldType.array(FieldType.STRING) matching the BigQuery REPEATED type.
Regenerate the Beam schema from the actual table schema with BigQueryUtils.fromTableSchema(...) so REPEATED columns become arrays.
If the field should be scalar, flatten the BigQuery value upstream (UNNEST or take [SAFE_ORDINAL]) before conversion.
Verify the target FieldType actually corresponds to the BigQuery column type for that field name.

Example fix

// before
Field f = Field.of("tags", FieldType.STRING); // BigQuery REPEATED<STRING>

// after
Field f = Field.of("tags", FieldType.array(FieldType.STRING));

Defensive patterns

Strategy: validation

Validate before calling

// Java: ensure every List-valued JSON field maps to an array FieldType
schema.getFields().forEach(f -> {
  Object v = tableRow.get(f.getName());
  if (v instanceof List && f.getType().getCollectionElementType() == null) {
    throw new IllegalArgumentException("Field must be array-typed: " + f.getName());
  }
});

Type guard

// Java
static boolean listValueHasArrayType(TableRow tr, Schema.Field f) {
  return !(tr.get(f.getName()) instanceof List) || f.getType().getCollectionElementType() != null;
}

Try / catch

try {
  Row row = BigQueryUtils.toBeamRow(schema, tableRow);
} catch (IllegalArgumentException e) {
  // inspect schema mismatch reported in message; route to dead-letter
}

Prevention

Map every REPEATED BigQuery column to FieldType.array(elementType) in the Beam schema.
Diff the Beam schema against the table schema at pipeline startup (fail fast on drift).
Beware of table schema changes turning scalar columns into ARRAY columns.

When it happens

Trigger: toBeamRow conversion encountering a repeated BigQuery column (JSON array value) mapped to a Beam FieldType that is not FieldType.array(...) (getCollectionElementType() == null).

Common situations: Hand-written Beam schema where a REPEATED BigQuery column was declared as a scalar; table schema changed a column from scalar to ARRAY after the pipeline schema was written; using a generic FieldType for a field that BigQuery returns as a list.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1dbfc52e5e8d9fe8. Report an issue: GitHub.