apache/beam · error · RuntimeException

Reserved field name <field.name()> in user schema.

Error message

Reserved field name <field.name()> in user schema.

What it means

When converting an Avro schema to a BigQuery Storage API TableSchema, fields whose names collide with StorageApiCDC reserved column names are rejected with RuntimeException 'Reserved field name X in user schema.' BigQuery CDC metadata columns must not be shadowed by user fields.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroGenericRecordToStorageApiProto.java:325

    if (changeType != null) {
      builder.setField(
          org.apache.beam.sdk.util.Preconditions.checkStateNotNull(
              descriptor.findFieldByName(StorageApiCDC.CHANGE_TYPE_COLUMN)),
          changeType);
      builder.setField(
          org.apache.beam.sdk.util.Preconditions.checkStateNotNull(
              descriptor.findFieldByName(StorageApiCDC.CHANGE_SQN_COLUMN)),
          org.apache.beam.sdk.util.Preconditions.checkStateNotNull(changeSequenceNum));
    }
    return builder.build();
  }

  private static TableFieldSchema fieldDescriptorFromAvroField(org.apache.avro.Schema.Field field) {
    @Nullable Schema schema = field.schema();

    Preconditions.checkNotNull(schema, "Unexpected null schema!");
    if (StorageApiCDC.COLUMNS.contains(field.name())) {
      throw new RuntimeException("Reserved field name " + field.name() + " in user schema.");
    }
    TableFieldSchema.Builder builder =
        TableFieldSchema.newBuilder().setName(field.name().toLowerCase());
    Schema elementType = null;
    switch (schema.getType()) {
      case RECORD:
        Preconditions.checkState(!schema.getFields().isEmpty());
        builder = builder.setType(TableFieldSchema.Type.STRUCT);
        for (Schema.Field recordField : schema.getFields()) {
          builder = builder.addFields(fieldDescriptorFromAvroField(recordField));
        }
        break;
      case ARRAY:
        elementType = TypeWithNullability.create(schema.getElementType()).getType();
        if (elementType == null) {
          throw new RuntimeException("Unexpected null element type!");
        }
        Preconditions.checkState(

View on GitHub (pinned to 12126d8942)

Solutions

1. Rename the Avro field so it no longer collides with StorageApiCDC reserved names
2. Remove the reserved field from the schema if it is not needed
3. Disable/move CDC usage so the column names are allowed

Example fix

// before
{"name": "<reserved_cdc_column>", "type": "string"}
// after
{"name": "user_<reserved_cdc_column>", "type": "string"} // renamed to avoid collision

for (org.apache.avro.Schema.Field f : avroSchema.getFields()) { if (StorageApiCDC.COLUMNS.contains(f.name())) throw new IllegalStateException("reserved field: " + f.name()); }

boolean hasNoCdcReservedNames(org.apache.avro.Schema s) { return s.getFields().stream().noneMatch(f -> StorageApiCDC.COLUMNS.contains(f.name())); }

try { protoTableSchema = AvroGenericRecordToStorageApiProto.protoTableSchemaFromAvroSchema(schema, true); } catch (RuntimeException e) { if (e.getMessage().contains("Reserved field name")) { renameAndRetry(); } else throw e; }

When it happens

Trigger: AvroGenericRecordToStorageApiProto.fieldDescriptorFromAvroField encountering an Avro field whose name is in StorageApiCDC.COLUMNS while building the table schema for a Storage Write API sink.

Common situations: Using a CDC-enabled BigQuery table while the source Avro schema contains columns like the CDC reserved names; schema drift after enabling StorageApiCDC.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• RECORD/STRUCT are not primitive types
• Unknown BigQuery type: " + bqType
• Unknown BigQuery Field Mode: %s
• Unknown Avro type: " + type.getType()
• Unsupported type <elementType.getType()>

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/087dc987ea4d875c. Report an issue: GitHub.