apache/beam · error · SchemaDoesntMatchException

Couldn't convert schema for

Error message

Couldn't convert schema for %s

What it means

TableRowToStorageApiProto wraps any exception raised while building the final protobuf DynamicMessage for a row into a SchemaDoesntMatchException, adding the full table name. It means the TableRow did not match the BigQuery table's schema at the top level (after per-field errors were collected). The underlying cause is chained as the exception's cause.

Solutions

1. Log the chained cause (e.getCause()) to find the exact failing field.
2. Validate that every non-nullable schema field is present and non-null in each TableRow.
3. Align nested TableRow maps with the declared schema (field names lowercased, matching RECORD types).
4. Re-read the table schema (bigquery.tables().get) and rebuild the Beam schema before writing.
5. Enable getNestedUnknown handling or update the pipeline schema to match actual data.

Example fix

// before
row.get("created_at") may be null for a REQUIRED field
// after
if (row.get("created_at") == null) { row.set("created_at", Instant.now().toString()); }

List<String> missing = schema.getFields().stream().filter(f -> !f.getMode().equals("NULLABLE") && row.get(f.getName().toLowerCase()) == null).map(Field::getName).collect(toList()); if (!missing.isEmpty()) throw new IllegalArgumentException("Missing required fields: " + missing);

boolean matchesSchema(Map<String,Object> row, TableSchema schema) { return schema.getFields().stream().allMatch(f -> row.containsKey(f.getName().toLowerCase()) || "NULLABLE".equals(f.getMode())); }

try { ProtoRow proto = TableRowToStorageApiProto.tableRowToProto(tableSchema, row); } catch (SchemaDoesntMatchException e) { LOG.error("Row rejected for table {}: cause={}", table, e.getCause(), e); deadLetter(row, e); }

When it happens

Trigger: Calling the BigQueryIO Storage Write API sink (via tableRowToStorageApiProto / messageFromTableRow) when one or more fields failed schema conversion, so the builder was never produced, or DynamicMessage.build() itself rejected an unset required field.

Common situations: Rows missing required (non-nullable) columns; nested row structures not matching the declared RECORD schema; value types (e.g. Long vs String for INT64) that converters reject; schema drift where the table was altered but the Beam schema or data wasn't updated.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Converting BigQuery type
• Could convert schema for
• Error while converting table schema
• Field is not nullable.
• Nested ROW missing row schema

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4bc8edf9628e9924. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:1044

              Preconditions.checkStateNotNull(changeSequenceNum));
    }

    if (requiredFieldsRemaining != null && !requiredFieldsRemaining.isEmpty()) {
      String prefix = schemaInformation.getFullName();
      Set<String> missingFields =
          requiredFieldsRemaining.stream()
              .map(key -> prefix.isEmpty() ? key : String.join(".", prefix, key))
              .collect(Collectors.toSet());
      SchemaConversionException e = new SchemaMissingRequiredFieldException(missingFields);
      collectedExceptions.collect(e); // Throws if not collected.
    }
    if (!collectedExceptions.isEmpty()) {
      return null;
    }
    try {
      return Preconditions.checkArgumentNotNull(builder).build();
    } catch (Exception e) {
      throw new SchemaDoesntMatchException(
          "Couldn't convert schema for " + schemaInformation.getFullName(), e);
    }
  }

  /**
   * Forwards {@code changeSequenceNum} to {@link #messageFromTableRow(SchemaInformation,
   * Descriptor, TableRow, boolean, boolean, TableRow, String, String, ErrorCollector)} via {@link
   * Long#toHexString}.
   */
  public static @Nullable DynamicMessage messageFromTableRow(
      SchemaInformation schemaInformation,
      @Nullable Descriptor descriptor,
      TableRow tableRow,
      boolean ignoreUnknownValues,
      boolean allowMissingRequiredFields,
      final @Nullable TableRow unknownFields,
      @Nullable String changeType,
      long changeSequenceNum,

View on GitHub (pinned to 12126d8942)