apache/beam · error · IllegalArgumentException

Cannot merge schemas with different numbers of fields. schem

Error message

Cannot merge schemas with different numbers of fields. schema1: +schema1+ schema2: +schema2

What it means

mergeWideningNullable merges two Schemas field-by-field, producing a schema whose field types are nullable if either input's corresponding field is nullable. It requires both schemas to have exactly the same number of fields; otherwise it throws IllegalArgumentException with both schemas rendered in the message. This is a fail-fast precondition check for schema evolution/merging.

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/SchemaUtils.java:44

import org.apache.beam.sdk.values.Row;

/** A set of utility functions for schemas. */
@SuppressWarnings({
  "nullness" // TODO(https://github.com/apache/beam/issues/20497)
})
public class SchemaUtils {
  private static final String INDENT = "  ";

  /**
   * Given two schema that have matching types, return a nullable-widened schema.
   *
   * <p>The schemas must have matching types, except for field names which can differ. The returned
   * schema will contain the field names in the first schema. All field types will be nullable if
   * the corresponding field type is nullable in either of the input schemas.
   */
  public static Schema mergeWideningNullable(Schema schema1, Schema schema2) {
    if (schema1.getFieldCount() != schema2.getFieldCount()) {
      throw new IllegalArgumentException(
          "Cannot merge schemas with different numbers of fields. "
              + "schema1: "
              + schema1
              + " schema2: "
              + schema2);
    }
    Schema.Builder builder = Schema.builder();
    for (int i = 0; i < schema1.getFieldCount(); ++i) {
      String name = schema1.getField(i).getName();
      builder.addField(
          name, widenNullableTypes(schema1.getField(i).getType(), schema2.getField(i).getType()));
    }
    return builder.build();
  }

  static FieldType widenNullableTypes(FieldType fieldType1, FieldType fieldType2) {
    if (fieldType1.getTypeName() != fieldType2.getTypeName()) {
      throw new IllegalArgumentException(

View on GitHub (pinned to 12126d8942)

Solutions

Ensure both schemas have the same field count before merging; add the missing field to the shorter schema.
If schemas legitimately diverged, use Schema.mergeSchemas / Schema.builder() to build a union schema instead of mergeWideningNullable.
Verify both schemas are generated from the same source-of-truth (same class/Avro file/version) and redeploy consistent code.
Catch IllegalArgumentException and log both schemas to identify which field was added/removed.

Example fix

// before
Schema merged = SchemaUtils.mergeWideningNullable(oldSchema, newSchema); // throws: newSchema has an extra field
// after
if (oldSchema.getFieldCount() == newSchema.getFieldCount()) {
  Schema merged = SchemaUtils.mergeWideningNullable(oldSchema, newSchema);
} else {
  Schema merged = SchemaUtils.mergeSchemas(oldSchema, newSchema); // union merge
}

Defensive patterns

Strategy: validation

Validate before calling

if (schema1.getFieldCount() != schema2.getFieldCount()) {
  throw new IllegalStateException("mergeWideningNullable precondition failed: "
      + schema1.getFieldCount() + " vs " + schema2.getFieldCount() + " fields");
}
Schema merged = SchemaUtils.mergeWideningNullable(schema1, schema2);

Type guard

boolean canWidenMerge(Schema s1, Schema s2) {
  return s1.getFieldCount() == s2.getFieldCount();
}

Try / catch

try {
  Schema merged = SchemaUtils.mergeWideningNullable(schema1, schema2);
} catch (IllegalArgumentException e) {
  LOG.error("Field count mismatch merging schemas: {}", e.getMessage());
  throw new SchemaMergeException(e);
}

Prevention

Generate both schemas from the same source definition and version them together.
Assert field-count equality in a unit test before any schema merge runs.
Prefer SchemaUtils.mergeSchemas when schemas may legitimately diverge.

When it happens

Trigger: Calling SchemaUtils.mergeWideningNullable(schema1, schema2) where schema1.getFieldCount() != schema2.getFieldCount() — e.g. one side of a merge added or dropped a field (schema evolution, codegen drift between pipeline versions).

Common situations: Recomputing an output schema after a Beam schema evolved (new column added upstream); comparing a generated schema (e.g. from Avro/POJO autogen) against a hand-written one; running an old pipeline definition against data whose schema was extended in a newer version.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/059daebc7f9121f8. Report an issue: GitHub.