apache/beam · error · IllegalArgumentException

Field is not nullable.

Error message

Field is not nullable.

What it means

Thrown by BigQueryUtils.fromBeamField when converting a Beam row field back toward a BigQuery/Avro representation and the value is null while the beam Schema FieldType is declared non-nullable. The Beam schema contract forbids nulls in required fields, so the library fails fast instead of silently writing a null into a REQUIRED column. This is a schema/data mismatch, not an infrastructure failure.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java:748

  /** Convert a Beam Row to a BigQuery TableRow. */
  public static TableRow toTableRow(Row row) {
    TableRow output = new TableRow();
    for (int i = 0; i < row.getFieldCount(); i++) {
      @Nullable Object value = row.getValue(i);
      Field schemaField = row.getSchema().getField(i);
      @SuppressWarnings("nullness") // TableRow.set is not annotated, but accepts nulls
      TableRow updated =
          output.set(schemaField.getName(), fromBeamField(schemaField.getType(), value));
      output = updated;
    }
    return output;
  }

  private static @Nullable Object fromBeamField(FieldType fieldType, @Nullable Object fieldValue) {
    if (fieldValue == null) {
      if (!fieldType.getNullable()) {
        throw new IllegalArgumentException("Field is not nullable.");
      }
      return null;
    }

    switch (fieldType.getTypeName()) {
      case ARRAY:
      case ITERABLE:
        FieldType elementType =
            Preconditions.checkArgumentNotNull(fieldType.getCollectionElementType());
        Iterable<?> items = (Iterable<?>) fieldValue;
        List<@Nullable Object> convertedItems =
            Lists.newArrayListWithCapacity(Iterables.size(items));
        for (Object item : items) {
          convertedItems.add(fromBeamField(elementType, item));
        }
        return convertedItems;

      case MAP:

View on GitHub (pinned to 12126d8942)

Solutions

  1. Declare the field NULLABLE in the Beam Schema (Schema.Field.of(name, FieldType.X.withNullable(true))) to match the data.
  2. Populate the field before conversion (e.g. withRow fallthrough/default value in the DoFn that builds the row).
  3. Filter out rows with nulls in required fields before the conversion step.
  4. If the BigQuery table column should accept nulls, change the BigQuery schema to NULLABLE and mirror it in the Beam schema.

Example fix

// before
Schema.Field f = Schema.Field.of("id", FieldType.INT64); // required
row.getInt("id") == null -> throws

// after
Schema.Field f = Schema.Field.of("id", FieldType.INT64.withNullable(true));
// or ensure a value: row = new RowBuilder().addInt("id", id == null ? -1 : id).build();
Defensive patterns

Strategy: validation

Validate before calling

// Java: validate row against schema before conversion
for (Schema.Field f : row.getSchema().getFields()) {
  if (!f.getType().getNullable() && row.getValue(f.getName()) == null) {
    throw new IllegalArgumentException("Non-nullable field has null value: " + f.getName());
  }
}

Type guard

// Java
static boolean isUsableRequiredValue(Row row, String name) {
  Schema.Field f = row.getSchema().getField(name);
  return f.getType().getNullable() || row.getValue(name) != null;
}

Try / catch

try {
  avroRecord = BigQueryUtils.toAvro(schema, row);
} catch (IllegalArgumentException e) {
  // log and route row to dead-letter PCollection
}

Prevention

When it happens

Trigger: Calling BeamRow -> Avro/BigQuery conversion (e.g. BigQueryUtils.toAvro or row-to-JSON paths) via fromBeamField with a null fieldValue on a FieldType whose getNullable() is false.

Common situations: Building TableRow/Avro records from Beam rows where a REQUIRED BigQuery column has no value; rows produced by a DoFn that leaves required fields unset; a schema evolved from NULLABLE to REQUIRED while old data still contains nulls.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7f12b8e5b3015625. Report an issue: GitHub.