apache/beam · error · IllegalArgumentException

Field not nullable

Error message

Field %s not nullable

What it means

Thrown by BigQueryUtils conversion from Avro to Beam Row (avroGenericRecordToBeamRow path) when the Avro GenericRecord value is null but the corresponding Beam FieldType is non-nullable. The message includes the Beam field type. This mirrors the JSON-side null check: required Beam fields cannot be built from Avro nulls, so the library throws IllegalArgumentException.

Solutions

1. Mark the field nullable in the Beam Schema (FieldType.withNullable(true)) since BigQuery Avro representations are typically nullable.
2. Fill defaults for null Avro values before conversion (record.put(field, default) or a Map/DoFn step).
3. Filter records containing nulls in required fields before avroGenericRecordToBeamRow.
4. Generate the Beam schema with fromAvroSchema / fromTableSchema so nullability derives from the source.

Example fix

// before
Field f = Field.of("user_id", FieldType.INT64); // Avro value often null

// after
Field f = Field.of("user_id", FieldType.INT64.withNullable(true));
// or pre-fill: if (record.get("user_id") == null) record.put("user_id", 0L);

// Java: validate Avro record nulls vs Beam schema before conversion
for (Schema.Field f : beamSchema.getFields()) {
  if (!f.getType().getNullable() && record.get(f.getName()) == null) {
    throw new IllegalArgumentException("Avro null in required field: " + f.getName());
  }
}

// Java
static boolean avroFieldNonNull(GenericRecord r, String name) {
  return r.get(name) != null;
}

try {
  Row row = BigQueryUtils.avroGenericRecordToBeamRow(beamSchema, options, record);
} catch (IllegalArgumentException e) {
  // dead-letter record; message names the offending field type
}

When it happens

Trigger: Calling BigQueryUtils.avroGenericRecordToBeamRow(schema, options, record) where record.get(fieldName) is null for a Beam field with nullable=false; typically when reading BigQuery export Avro files whose column is nullable at the Avro level but the Beam schema marks it REQUIRED.

Common situations: Reading BigQuery Storage Read / export-to-GCS Avro files where all columns are Avro-nullable, while the Beam schema (e.g. generated from a REQUIRED table schema or hand-written) expects non-null; legacy rows written before a column became REQUIRED.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Field is not nullable.
• Received null value for non-nullable field ""
• Received null value for non-nullable field " +…
• Received null value for non-nullable field " +…
• RECORD/STRUCT are not primitive types

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6f194b199b68bb3c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtils.java:1019

  /**
   * Tries to convert an Avro decoded value to a Beam field value based on the target type of the
   * Beam field.
   *
   * <p>For the Avro formats of BigQuery types, see
   * https://cloud.google.com/bigquery/docs/exporting-data#avro_export_details and
   * https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#avro_conversions
   */
  public static @Nullable Object convertAvroFormat(
      FieldType beamFieldType,
      @Nullable Object avroValue,
      BigQueryUtils.ConversionOptions options) {
    TypeName beamFieldTypeName = beamFieldType.getTypeName();
    if (avroValue == null) {
      if (beamFieldType.getNullable()) {
        return null;
      } else {
        throw new IllegalArgumentException(String.format("Field %s not nullable", beamFieldType));
      }
    }
    switch (beamFieldTypeName) {
      case BYTE:
      case INT16:
      case INT32:
      case INT64:
      case FLOAT:
      case DOUBLE:
      case STRING:
      case BYTES:
      case BOOLEAN:
        return convertAvroPrimitiveTypes(beamFieldTypeName, avroValue);
      case DATETIME:
        // Expecting value in microseconds.
        switch (options.getTruncateTimestamps()) {
          case TRUNCATE:
            return truncateToMillis(avroValue);

View on GitHub (pinned to 12126d8942)