apache/beam · error · RuntimeException

Reserved field name " + field.getName() + " in user schema.

Error message

Reserved field name " + field.getName() + " in user schema.

What it means

fieldDescriptorFromBeamField rejects Beam Schema fields whose names collide with the reserved Storage API CDC columns (StorageApiCDC.COLUMNS, e.g. "record_number", "replay", "modified_time"). Since BigQuery appends CDC metadata columns internally, a user field with such a name would conflict, so a RuntimeException is thrown.

Solutions

1. Rename the field in your Beam Schema to something not in StorageApiCDC.COLUMNS (e.g. recordNumber or record_num)
2. Rename the column at the data source level before building the schema
3. Check StorageApiCDC.COLUMNS in the Beam source to see the exact reserved names
4. If renaming is impossible, write to a non-CDC sink or stage the data through a rename step

Example fix

// before
Schema schema = Schema.builder().addInt64Field("record_number").build();
// after
Schema schema = Schema.builder().addInt64Field("record_num").build();

// Java
for (Schema.Field f : beamSchema.getFields()) {
  if (StorageApiCDC.COLUMNS.contains(f.getName())) {
    throw new IllegalStateException("Reserved CDC column name: " + f.getName());
  }
}

// Java
boolean reserved = StorageApiCDC.COLUMNS.contains(fieldName); // check before adding field

// Java
try {
  schema = BeamRowToStorageApiProto.protoTableSchemaFromBeamSchema(beamSchema);
} catch (RuntimeException e) {
  if (e.getMessage().startsWith("Reserved field name")) {
    // rename offending column and rebuild schema
  }
  throw e;
}

When it happens

Trigger: Writing a Beam Row schema to BigQuery via Storage API (especially with CDC enabled) where any field — including nested fields — is named like a reserved CDC column

Common situations: Schemas derived from external tables that happen to have a column named record_number or modified_time; auto-generated schemas from JSON/CSV headers containing these names.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Cannot cast to a compatible object to build ByteString.
• Unexpected null element type on " + field.getName()
• Unexpected null logical type " + field.getType()
• Unexpected null schema!
• A function must be provided to convert the input type into…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/93b76789c564e20a. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BeamRowToStorageApiProto.java:206

    }
    return builder.build();
  }

  @VisibleForTesting
  static TableSchema protoTableSchemaFromBeamSchema(Schema schema) {
    Preconditions.checkState(schema.getFieldCount() > 0);

    TableSchema.Builder builder = TableSchema.newBuilder();
    for (Field field : schema.getFields()) {
      builder.addFields(fieldDescriptorFromBeamField(field));
    }
    return builder.build();
  }

  private static TableFieldSchema fieldDescriptorFromBeamField(Field field) {
    TableFieldSchema.Builder builder = TableFieldSchema.newBuilder();
    if (StorageApiCDC.COLUMNS.contains(field.getName())) {
      throw new RuntimeException("Reserved field name " + field.getName() + " in user schema.");
    }
    builder = builder.setName(field.getName().toLowerCase());

    switch (field.getType().getTypeName()) {
      case ROW:
        @Nullable Schema rowSchema = field.getType().getRowSchema();
        if (rowSchema == null) {
          throw new RuntimeException("Unexpected null schema!");
        }
        builder = builder.setType(TableFieldSchema.Type.STRUCT);
        for (Schema.Field nestedField : rowSchema.getFields()) {
          builder = builder.addFields(fieldDescriptorFromBeamField(nestedField));
        }
        break;
      case ARRAY:
      case ITERABLE:
        @Nullable FieldType elementType = field.getType().getCollectionElementType();
        if (elementType == null) {

View on GitHub (pinned to 12126d8942)