apache/beam · error · IllegalArgumentException

Can't convert 'null' to non-nullable field

Error message

Can't convert 'null' to non-nullable field

What it means

AvroUtils.convertAvroFieldStrict performs strict Avro-to-Beam Schema value conversion when materializing GenericRecords into Beam Rows. When the Avro schema branch being converted is Schema.Type.NULL (i.e., the value is literally null and the target Beam FieldType is not nullable), the converter cannot produce a non-null value for a non-nullable Beam field and throws this IllegalArgumentException. It exists because strict conversion refuses to silently coerce null into a field the Beam schema declares as non-nullable.

Source

Thrown at sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java:1591

      case ENUM:
        // enums are either Java enums, or GenericEnumSymbol,
        // they don't share common interface, but override toString()
        return convertEnumStrict(value, fieldType);

      case ARRAY:
        return convertArrayStrict(
            (List<Object>) value, type.type.getElementType(), fieldType, genericData);

      case MAP:
        return convertMapStrict(
            (Map<CharSequence, Object>) value, type.type.getValueType(), fieldType, genericData);

      case UNION:
        return convertUnionStrict(value, type.type, fieldType, genericData);

      case NULL:
        throw new IllegalArgumentException("Can't convert 'null' to non-nullable field");

      default:
        throw new AssertionError("Unexpected AVRO Schema.Type: " + type.type.getType());
    }
  }

  /**
   * Strict conversion from AVRO to Beam, strict because it doesn't do widening or narrowing during
   * conversion.
   *
   * @param value {@link GenericRecord} or any nested value
   * @param avroSchema schema for value
   * @param fieldType target beam field type
   * @return value converted for {@link Row}
   */
  @SuppressWarnings("unchecked")
  public static @PolyNull Object convertAvroFieldStrict(
      @PolyNull Object value,

View on GitHub (pinned to 12126d8942)

Solutions

1. Make the corresponding Beam Schema field nullable (FieldType nullable: fieldType.withNullable(true) or "field": FieldType.STRING.withNullable(true)) so null values can be represented.
2. Sanitize/replace null values before conversion (e.g., via a MapElements that substitutes defaults) so non-nullable fields never receive null.
3. Verify the Beam schema was generated to match the Avro schema (use AvroUtils.toBeamSchema(avroSchema) instead of a hand-written schema) so nullability matches.

Example fix

// before
Schema beamSchema = Schema.of(Field.of("name", FieldType.STRING));
Row row = toBeamRowStrict(avroRecord, beamSchema); // throws when name is null

// after
Schema beamSchema = Schema.of(Field.of("name", FieldType.STRING.withNullable(true)));

Schema beamSchema = AvroUtils.toBeamSchema(avroSchema);
for (Field f : beamSchema.getFields()) {
  Object v = record.get(f.getName());
  if (v == null && !f.getType().getNullable()) {
    throw new IllegalStateException("field " + f.getName() + " is null but non-nullable");
  }
}

try {
  Row row = AvroUtils.toBeamRowStrict(record, beamSchema);
} catch (IllegalArgumentException e) {
  if (e.getMessage().contains("non-nullable")) {
    // handle null-in-non-nullable field (default or skip)
  } else { throw e; }
}

When it happens

Trigger: Calling convertAvroFieldStrict(value, avroSchema, fieldType) (or toBeamRowStrict via AvroRecorder/AvroIO reads) where the resolved union branch or schema type is NULL but the target Beam FieldType is non-nullable — i.e., an Avro schema declares "null" only in a union (e.g. ["null","string"]) yet the Beam schema field was built non-nullable, or a GenericRecord holds a null for a field whose Beam FieldType lacks the NULLABLE flag.

Common situations: Reading Avro files whose schema has nullable union fields (the default Avro idiom) into a Beam schema defined by hand without nullable fields; a schema drift where the Avro file contains nulls that the Beam schema does not allow; building Row objects in tests with null values for non-nullable fields.

Related errors

• Can't convert 'null' to FieldType
• Unhandled logical type ${identifier}
• Avro only supports maps with string keys
• Unexpected type ${fieldType}
• Can't convert 'string' map keys to ${mapKeyType}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a2df699d9ace2d4f. Report an issue: GitHub.