apache/beam · error · IllegalArgumentException

Received null value for non-nullable field " + fieldDescript

Error message

Received null value for non-nullable field " + fieldDescriptor.getName()

What it means

messageValueFromGenericRecordValue throws IllegalArgumentException when a null value arrives for a BigQuery Storage API field that is neither optional in the proto descriptor nor nullable in the Avro union schema - i.e. a required field received null.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroGenericRecordToStorageApiProto.java:461

      }
    }
    if (field.doc() != null) {
      builder = builder.setDescription(field.doc());
    }
    return builder.build();
  }

  @Nullable
  private static Object messageValueFromGenericRecordValue(
      FieldDescriptor fieldDescriptor, Schema.Field avroField, String name, GenericRecord record) {
    @Nullable Object value = record.get(name);
    if (value == null) {
      if (fieldDescriptor.isOptional()
          || avroField.schema().getTypes().stream()
              .anyMatch(t -> t.getType() == Schema.Type.NULL)) {
        return null;
      } else {
        throw new IllegalArgumentException(
            "Received null value for non-nullable field " + fieldDescriptor.getName());
      }
    }
    return toProtoValue(fieldDescriptor, avroField.schema(), value);
  }

  private static Object toProtoValue(
      FieldDescriptor fieldDescriptor, Schema avroSchema, Object value) {
    switch (avroSchema.getType()) {
      case RECORD:
        return messageFromGenericRecord(
            fieldDescriptor.getMessageType(), (GenericRecord) value, null, -1);
      case ARRAY:
        Iterable<Object> iterable = (Iterable<Object>) value;
        @Nullable Schema arrayElementType = avroSchema.getElementType();
        if (arrayElementType == null) {
          throw new RuntimeException("Unexpected null element type!");
        }

View on GitHub (pinned to 12126d8942)

Solutions

1. Declare the BigQuery column NULLABLE (or mark the Avro field as ["null","type"])
2. Cleanse/replace null values with defaults before writing
3. Coerce upstream producers to always populate the field

Example fix

// before
{"name":"id","type":"long"} // non-nullable, but records may contain null
// after
{"name":"id","type":["null","long"],"default":null} // or filter null records before the sink

if (record.get(fieldName) == null && fieldMode == Field.Mode.REQUIRED) throw new IllegalStateException("null for REQUIRED field " + fieldName);

boolean nullSafeForBq(org.apache.avro.generic.GenericRecord r, org.apache.avro.Schema.Field f) { return r.get(f.name()) != null || f.schema().getTypes().stream().anyMatch(t -> t.getType() == org.apache.avro.Schema.Type.NULL); }

try { proto = AvroGenericRecordToStorageApiProto.messageValueFromGenericRecordValue(fieldDescriptor, avroField, record); } catch (IllegalArgumentException e) { if (e.getMessage().contains("non-nullable field")) { applyDefaultOrDrop(); } else throw e; }

When it happens

Trigger: Converting a GenericRecord to proto where a field is null, the TableFieldSchema is not OPTIONAL (REQUIRED mode), and the Avro field's union does not include NULL.

Common situations: Upstream data producing nulls for columns declared REQUIRED in BigQuery; Avro schema says non-nullable but actual records contain nulls (schema/data drift); backfills of legacy data.

Related errors

• Reserved field name <field.name()> in user schema.
• Unsupported type <elementType.getType()>
• RECORD/STRUCT are not primitive types
• Unknown BigQuery type: " + bqType
• Unknown BigQuery Field Mode: %s

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ab15e34d95c9dc65. Report an issue: GitHub.