apache/beam · error · IllegalArgumentException

Error converting field :

Error message

Error converting field : 

What it means

toBeamRow wraps any exception thrown while converting one Avro record field to its Beam value in an IllegalArgumentException with the message "Error converting field <field>: <cause>". The original cause (NumberFormatException, ClassCastException, Avro type mismatch, etc.) is preserved as the cause. It indicates the Avro record read from BigQuery does not match the expected Beam Schema field during the Avro-format to Beam Row conversion.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java:706

    @Override
    public TableRow apply(T input) {
      return toTableRow(toRow.apply(input));
    }
  }

  public static Row toBeamRow(GenericRecord record, Schema schema, ConversionOptions options) {
    List<@Nullable Object> valuesInOrder =
        schema.getFields().stream()
            .<@Nullable Object>map(
                field -> {
                  try {
                    org.apache.avro.Schema.Field avroField =
                        record.getSchema().getField(field.getName());
                    @Nullable Object value = avroField != null ? record.get(avroField.pos()) : null;
                    return convertAvroFormat(field.getType(), value, options);
                  } catch (Exception cause) {
                    throw new IllegalArgumentException(
                        "Error converting field " + field + ": " + cause.getMessage(), cause);
                  }
                })
            .collect(toList());

    return Row.withSchema(schema).addValues(valuesInOrder).build();
  }

  /**
   * Convert generic record to Bq TableRow.
   *
   * @deprecated use {@link #convertGenericRecordToTableRow(GenericRecord)}
   */
  @Deprecated
  public static TableRow convertGenericRecordToTableRow(
      GenericRecord record, TableSchema tableSchema) {
    return convertGenericRecordToTableRow(record);
  }

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the chained cause (IllegalArgumentException#getCause) to find the actual failing conversion and fix the offending data or field mapping.
2. Re-derive the Beam Schema from the actual BigQuery table (BigQueryUtils.fromTableSchema / BeamIO's schema inference) instead of a hand-written schema so it matches the Avro export.
3. Null-out or widen the problematic field in the schema (e.g. make it NULLABLE or STRING) and cast/convert afterwards with map/ParDo logic you control.
4. Check for schema drift: compare the current table schema against the pipeline's expectations and update/redeploy accordingly.

Example fix

// before (hand-written, drifted schema)
Schema schema = Schema.builder().addField("amount", FieldType.INT64).build();
// after (infer from table)
Schema schema = BigQueryUtils.fromTableSchema(bqClient.getTable(tableId).getDefinition().getSchema());

// compare expected schema with the actual BigQuery table before reading
Table bqTable = bigquery.getTable(tableId);
Schema actual = BigQueryUtils.fromTableSchema(bqTable.getDefinition().getSchema());
if (!actual.equivalent(expectedSchema)) {
  throw new IllegalStateException("Schema drift detected between pipeline and BigQuery table");
}

static boolean rowMatchesSchema(org.apache.avro.generic.GenericRecord rec, Schema beamSchema) {
  for (Field f : beamSchema.getFields()) {
    org.apache.avro.Schema.Field af = rec.getSchema().getField(f.getName());
    if (af == null && !f.getType().getNullable()) return false;
  }
  return true;
}

try {
  pipeline.apply(BigQueryIO.readTableRows().withMethod(DIRECT_READ)...);
} catch (IllegalArgumentException e) {
  if (String.valueOf(e.getMessage()).startsWith("Error converting field")) {
    log.error("Avro->Beam row conversion failed; check schema drift. Cause:", e.getCause());
    // route to dead-letter / re-read with inferred schema
  } else { throw e; }
}

When it happens

Trigger: Reading BigQuery with storage API / Avro format (convertAvroFormat) when a record's field value cannot be converted to the Beam Schema field type: e.g. Avro union branch mismatch, null in a non-nullable field, numeric overflow, string where bytes expected, or schema drift between the BigQuery table and the expected schema.

Common situations: BigQuery table schema changed after the pipeline was built; reading with a user-provided Beam Schema that doesn't match the exported Avro schema; corrupt or legacy Avro rows; DATETIME/TIMESTAMP/NUMERIC values outside what the converter's parse logic accepts.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• Does not support converting unknown type value: " + beamFiel
• is not primitive type.
• Can't convert 'string' map keys to ${mapKeyType}
• Value ${value} of class ${valueClass} is not a supported typ
• Unexpected beam type " + fieldSchema

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5e35223fcb464666. Report an issue: GitHub.