apache/beam · error · IllegalArgumentException

FieldType and AVRO schema don't have matching nullability

Error message

FieldType ${fieldType} and AVRO schema ${avroSchema} don't have matching nullability

What it means

Thrown by AvroUtils.genericFromBeamField when converting a Beam Row field to an Avro GenericRecord value: the Beam FieldType's nullability (fieldType.getNullable()) does not match whether the target Avro schema is nullable (an unwrapped UNION with NULL). The library requires both sides to agree on nullability so the conversion stays type-safe and nulls are handled symmetrically.

Solutions

1. Align the Avro schema with the Beam FieldType nullability: wrap the Avro type in a union ["null", T] if the Beam field is nullable, or remove the union if it is not
2. Or make the Beam FieldType match the Avro schema: use Schema.FieldType.withNullable(true/false) on the offending field before conversion
3. Regenerate the Avro schema from the Beam schema (AvroUtils.toAvroSchema) instead of maintaining it by hand, so nullability cannot drift
4. If the schema came from a stored .avsc file, update it to reflect the current Beam schema

Example fix

// before: Beam field nullable, Avro schema not
Schema.FieldType fieldType = Schema.FieldType.STRING.withNullable(true);
org.apache.avro.Schema avroSchema = org.apache.avro.Schema.create(Type.STRING); // no null branch
// after
org.apache.avro.Schema avroSchema = org.apache.avro.Schema.createUnion(
    org.apache.avro.Schema.create(Type.NULL), org.apache.avro.Schema.create(Type.STRING));

boolean nullabilityMatches(Schema.FieldType beamField, org.apache.avro.Schema avroSchema) {
  boolean avroNullable = avroSchema.getType() == org.apache.avro.Schema.Type.UNION
      && avroSchema.getTypes().stream().anyMatch(t -> t.getType() == org.apache.avro.Schema.Type.NULL);
  return beamField.getNullable() == avroNullable;
}

if (!nullabilityMatches(fieldType, avroSchema)) {
  throw new IllegalStateException("Fix schema nullability before conversion");
}

try {
  GenericRecord record = AvroUtils.toGenericRecord(row, avroSchema);
} catch (IllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().contains("don't have matching nullability")) {
    avroSchema = AvroUtils.toAvroSchema(beamSchema); // regenerate consistent schema
  } else { throw e; }
}

When it happens

Trigger: Calling toGenericRecord / toAvroType (directly or via AvroUtils.toGenericRecord or schema conversion APIs) with a Beam schema field and an Avro schema whose nullability disagree — e.g. a nullable Beam FieldType mapped to a plain (non-union) Avro schema, or a non-nullable Beam FieldType mapped to an Avro UNION ['null', T].

Common situations: Hand-written Avro schemas that forgot (or wrongly added) the ['null', T] union; Avro schemas regenerated after changing a Beam field to/from nullable; registerAvroSchema / Schemadrogel conversions where Beam schema was inferred from a Java class whose field nullability diverged from a stored .avsc file; version upgrades where one side's schema drifted.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Can't represent as
• Incorrectly sized byte array.
• Unsupported type
• Field not nullable
• Unexpected Avro field schema type

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5a9a91196c676dbf. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java:1283

    }
    return fieldType.getNullable() ? ReflectData.makeNullable(baseType) : baseType;
  }

  private static final Map<org.apache.avro.Schema, Function<Number, ? extends Number>>
      NUMERIC_CONVERTERS =
          ImmutableMap.of(
              org.apache.avro.Schema.create(org.apache.avro.Schema.Type.INT), Number::intValue,
              org.apache.avro.Schema.create(org.apache.avro.Schema.Type.LONG), Number::longValue,
              org.apache.avro.Schema.create(org.apache.avro.Schema.Type.FLOAT), Number::floatValue,
              org.apache.avro.Schema.create(org.apache.avro.Schema.Type.DOUBLE),
                  Number::doubleValue);

  /** Convert a value from Beam Row to a vlue used for Avro GenericRecord. */
  private static @Nullable Object genericFromBeamField(
      FieldType fieldType, org.apache.avro.Schema avroSchema, @Nullable Object value) {
    TypeWithNullability typeWithNullability = new TypeWithNullability(avroSchema);
    if (fieldType.getNullable() != typeWithNullability.nullable) {
      throw new IllegalArgumentException(
          "FieldType "
              + fieldType
              + " and AVRO schema "
              + avroSchema
              + " don't have matching nullability");
    }

    if (value == null) {
      return value;
    }

    if (NUMERIC_CONVERTERS.containsKey(typeWithNullability.type)) {
      return NUMERIC_CONVERTERS.get(typeWithNullability.type).apply((Number) value);
    }

    // TODO: should we use Avro Schema as the source-of-truth in general?
    switch (fieldType.getTypeName()) {
      case BYTE:

View on GitHub (pinned to 12126d8942)