apache/beam · error · SchemaDoesntMatchException

Inconsistent types seen for field

Error message

Inconsistent types seen for field: ${e.getMissingField()} ${oldValue.getType()} v.s. ${type}

What it means

UpgradeTableSchema.getIncrementalSchema merges field schemas encountered across elements into a single incremental schema. Duplicate fields are tolerated, but if the same field is seen with two different types it throws SchemaDoesntMatchException describing the field and the conflicting types, since BigQuery schema evolution cannot reconcile them.

Solutions

Make the field's type consistent across all elements — coerce values to one type before writing
Explicitly declare the field's type in the BigQuery schema so inference can't diverge
Normalize numeric values (e.g. always emit Double or always Long) in your TableRow construction
Split heterogeneous elements into separate PCollections/destinations with their own schemas
Log or quarantine the offending records where e.getMissingField() has inconsistent values

Example fix

// before
row.set("amount", maybeIntValue); // sometimes Long, sometimes Double
// after
row.set("amount", ((Number) maybeIntValue).doubleValue()); // always FLOAT type

Defensive patterns

Strategy: validation

Validate before calling

// normalize field types before writing
Object v = row.get(fieldName);
if (v instanceof Number) row.set(fieldName, ((Number) v).doubleValue()); // force FLOAT consistently

Type guard

Object coerceConsistentType(Object v, Class<?> expected) {
  if (v == null) return null;
  if (expected == Double.class && v instanceof Number) return ((Number) v).doubleValue();
  if (expected == Long.class && v instanceof Number) return ((Number) v).longValue();
  if (expected == String.class) return String.valueOf(v);
  return v;
}

Try / catch

try {
  upgradedSchema = UpgradeTableSchema.getIncrementalSchema(...);
} catch (TableRowToStorageApiProto.SchemaDoesntMatchException e) {
  LOG.error("Inconsistent type for field {}: {}", e.getMissingField(), e.getMessage());
  // coerce offending field or route record to DLQ
}

Prevention

Coerce numeric values to a single Java type when building TableRows
Declare explicit field types in the BigQuery schema instead of relying on inference
Keep record shapes homogeneous within a destination
Validate mixed-type fields (JSON sources) before writing

When it happens

Trigger: Processing elements where the same missing field is filled with different BigQuery types (e.g. STRING vs INTEGER) — often because values vary in Java type across bundle elements, causing type inference to derive different TableFieldSchema types.

Common situations: Heterogeneous records in one PCollection (polymorphic rows), a field sometimes null/absent and typed differently when present, numeric values sometimes inferred as INTEGER and sometimes as FLOAT, or JSON-sourced rows with mixed-type columns.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1d0ea02af42bc430. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/UpgradeTableSchema.java:93

            e.isRepeated() ? TableFieldSchema.Mode.REPEATED : TableFieldSchema.Mode.NULLABLE;
        // TODO(reuvenlax): Fix this so that arbitrary types can be selected.
        TableFieldSchema.Type type =
            e.isStruct() ? TableFieldSchema.Type.STRUCT : TableFieldSchema.Type.STRING;
        @Nullable TableFieldSchema oldValue =
            newFields
                .computeIfAbsent(prefix, p -> Maps.newLinkedHashMap())
                .put(
                    name,
                    TableFieldSchema.newBuilder()
                        .setName(name)
                        .setMode(mode)
                        .setType(type)
                        .build());
        if (oldValue != null) {
          // Duplicates are ok because we might run this over an entire bundle. However we must
          // ensure that they are compatible.
          if (!oldValue.getType().equals(type)) {
            throw new TableRowToStorageApiProto.SchemaDoesntMatchException(
                "Inconsistent types seen for field: "
                    + e.getMissingField()
                    + " "
                    + oldValue.getType()
                    + " v.s. "
                    + type);
          }
        }
      } else if (schemaConversionException
          instanceof TableRowToStorageApiProto.SchemaMissingRequiredFieldException) {
        ((TableRowToStorageApiProto.SchemaMissingRequiredFieldException) schemaConversionException)
            .getMissingFields()
            .forEach(
                f -> {
                  List<String> components = Arrays.asList(f.toLowerCase().split("\\."));
                  String prefix = String.join(".", components.subList(0, components.size() - 1));
                  String name = components.get(components.size() - 1);
                  relaxedFields.computeIfAbsent(prefix, p -> Sets.newHashSet()).add(name);

View on GitHub (pinned to 12126d8942)