apache/beam · error · IllegalArgumentException

AVRO schema doesn't match row schema. Row schema ${beamSchem

Error message

AVRO schema doesn't match row schema. Row schema ${beamSchema}. AVRO schema + ${avroSchema}

What it means

AvroUtils.toGenericRecord converts a Beam Row to an Avro GenericRecord. When an explicit AVRO schema is supplied, it requires that the number of top-level fields exactly equal the Beam row schema's field count; otherwise the record could not be mapped field-by-field. A mismatch throws this IllegalArgumentException naming both schemas.

Source

Thrown at sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/schemas/utils/AvroUtils.java:651

  /**
   * Convert from a Beam Row to an AVRO GenericRecord. The Avro Schema is inferred from the Beam
   * schema on the row.
   */
  public static GenericRecord toGenericRecord(Row row) {
    return toGenericRecord(row, null);
  }

  /**
   * Convert from a Beam Row to an AVRO GenericRecord. If a Schema is not provided, one is inferred
   * from the Beam schema on the row.
   */
  public static GenericRecord toGenericRecord(
      Row row, org.apache.avro.@Nullable Schema avroSchema) {
    Schema beamSchema = row.getSchema();
    // Use the provided AVRO schema if present, otherwise infer an AVRO schema from the row
    // schema.
    if (avroSchema != null && avroSchema.getFields().size() != beamSchema.getFieldCount()) {
      throw new IllegalArgumentException(
          "AVRO schema doesn't match row schema. Row schema "
              + beamSchema
              + ". AVRO schema + "
              + avroSchema);
    }
    if (avroSchema == null) {
      avroSchema = toAvroSchema(beamSchema);
    }

    GenericRecordBuilder builder = new GenericRecordBuilder(avroSchema);
    for (int i = 0; i < beamSchema.getFieldCount(); ++i) {
      Field field = beamSchema.getField(i);
      NullnessCheckerWorkarounds.builderSet(
          builder,
          field.getName(),
          genericFromBeamField(
              field.getType(), avroSchema.getField(field.getName()).schema(), row.getValue(i)));
    }

View on GitHub (pinned to 12126d8942)

Solutions

Regenerate or update the AVRO schema from the current Beam schema (AvroUtils.toAvroSchema(beamSchema)) and pass that instead.
Pass null as the AVRO schema so toGenericRecord infers a matching schema from the row schema.
Align the Beam schema: add/remove fields so the field count matches the AVRO schema.
If schemas legitimately evolved, map fields explicitly (build the GenericRecord manually or project the Row) before conversion.

Example fix

// before
GenericRecord rec = AvroUtils.toGenericRecord(row, staleAvroSchema);
// after
Schema avroSchema = AvroUtils.toAvroSchema(row.getSchema());
GenericRecord rec = AvroUtils.toGenericRecord(row, avroSchema);

Defensive patterns

Strategy: validation

Validate before calling

Schema beamSchema = row.getSchema();
if (avroSchema != null && avroSchema.getFields().size() != beamSchema.getFieldCount()) {
  avroSchema = AvroUtils.toAvroSchema(beamSchema); // re-derive matching schema
}
GenericRecord rec = AvroUtils.toGenericRecord(row, avroSchema);

Try / catch

try {
  return AvroUtils.toGenericRecord(row, avroSchema);
} catch (IllegalArgumentException e) {
  throw new SchemaMismatchException("Re-derive AVRO schema from beam schema", e);
}

Prevention

Never cache AVRO schemas across Beam schema changes; derive them via AvroUtils.toAvroSchema.
Run a schema-compatibility check (field count/names/types) in unit tests for schema-evolving pipelines.
Pass null to let the library infer the schema when you do not need a specific one.
Track schema versions explicitly when using a schema registry.

When it happens

Trigger: Calling AvroUtils.toGenericRecord(row, avroSchema) (directly or via genericFromBeamField / the getRowToAvroFunction path) where avroSchema.getFields().size() != row.getSchema().getFieldCount().

Common situations: Passing a stale Avro schema after adding/removing a Beam schema field; schema evolution where the writer's Avro schema is one revision older than the pipeline's inferred Beam schema; mixing a specific record schema with a differently-shaped row (e.g. nested rows flattened differently).

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6fde2bd46799d882. Report an issue: GitHub.