apache/beam · error · IllegalArgumentException

Received null value for non-nullable field " +…

Error message

Received null value for non-nullable field " + fieldDescriptor.getName()

What it means

messageValueFromRowValue converts a Row cell into a protobuf value for the Storage Write path. If the value is null but the target FieldDescriptor is neither optional (nullable) nor repeated, the row violates the schema's nullability and an IllegalArgumentException is thrown naming the field. BigQuery-required fields cannot receive null.

Solutions

1. Declare the field nullable: FieldType.string().withNullable(true), so null maps to an unset proto field.
2. Fill nulls before writing: use a DoFn/MapElements to replace null with a default (empty string, 0, sentinel).
3. Filter out rows that violate nullability before the sink.
4. Ensure the schema used for the Row matches the schema used to build the TableSchema (same generated class version).

Example fix

// before
Schema.Field f = Schema.Field.of("name", Schema.FieldType.STRING); // non-null
// after
Schema.Field f = Schema.Field.of("name", Schema.FieldType.STRING.withNullable(true));

for (Schema.Field f : schema.getFields()) {
  if (!f.getType().getNullable() && row.getValue(f.getName()) == null) {
    throw new IllegalArgumentException("Row field " + f.getName() + " is null but schema marks it non-nullable");
  }
}

boolean satisfiesNullability(Row row, Schema schema) {
  return schema.getFields().stream()
      .noneMatch(f -> !f.getType().getNullable() && row.getValue(f.getName()) == null);
}

try {
  DynamicMessage msg = messageFromBeamRow(descriptor, schema, row, unknownFields, -1);
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("Received null value for non-nullable field")) {
    row = fillDefaults(row);
    // retry once
  } else { throw e; }
}

When it happens

Trigger: Writing a Row via messageFromBeamRow/BeamRowToStorageApiProto where a field declared non-nullable in the Beam schema (getNullable()==false) contains an actual null value - e.g. a missing cell in a partially-filled row.

Common situations: Rows built via Row.withFieldNames without values for every field; upstream joins/coGroup producing nulls; data source with missing values while schema declares REQUIRED.

Related errors

• Field is not nullable.
• Field not nullable
• Received null value for non-nullable field ""
• A function must be provided to convert the input type into…
• BigQueryIO.Write transforms cannot be converted to a…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/dea2242c22c14f9b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BeamRowToStorageApiProto.java:322

      }
    }
    if (field.getDescription() != null) {
      builder = builder.setDescription(field.getDescription());
    }
    return builder.build();
  }

  @Nullable
  private static Object messageValueFromRowValue(
      FieldDescriptor fieldDescriptor, Field beamField, int index, Row row) {
    @Nullable Object value = row.getValue(index);
    if (value == null) {
      if (fieldDescriptor.isOptional()) {
        return null;
      } else if (fieldDescriptor.isRepeated()) {
        return Collections.emptyList();
      } else {
        throw new IllegalArgumentException(
            "Received null value for non-nullable field " + fieldDescriptor.getName());
      }
    }
    return toProtoValue(fieldDescriptor, beamField.getType(), value);
  }

  private static Object toProtoValue(
      FieldDescriptor fieldDescriptor, FieldType beamFieldType, Object value) {
    switch (beamFieldType.getTypeName()) {
      case ROW:
        return messageFromBeamRow(fieldDescriptor.getMessageType(), (Row) value, null, -1);
      case ARRAY:
      case ITERABLE:
        Iterable<Object> iterable = (Iterable<Object>) value;
        @Nullable FieldType iterableElementType = beamFieldType.getCollectionElementType();
        if (iterableElementType == null) {
          throw new RuntimeException("Unexpected null element type: " + fieldDescriptor.getName());
        }

View on GitHub (pinned to 12126d8942)