apache/beam · error · RuntimeException

Reserved field name in user schema.

Error message

Reserved field name %s in user schema.

What it means

BigQuery Storage API CDC reserves certain column names (e.g. StorageApiCDC.COLUMNS). A user-supplied TableFieldSchema using one of these reserved names cannot be converted into a proto descriptor and fails fast with a RuntimeException.

Solutions

1. Rename the offending column in your schema and data to something not in StorageApiCDC.COLUMNS.
2. Alias the column during extract/transform (e.g. withColumnRenamed / select with an alias).
3. Disable CDC (disableStreamingSideInputs / no-CDC write mode) if you cannot rename and don't need it.

Example fix

// before
TableFieldSchema.newBuilder().setName("record_name")...
// after
TableFieldSchema.newBuilder().setName("source_record_name")...

if (StorageApiCDC.COLUMNS.contains(fieldName)) { throw new IllegalArgumentException("Field " + fieldName + " collides with reserved CDC column"); }

boolean isAllowedUserField(String name) { return !StorageApiCDC.COLUMNS.contains(name); }

try { buildDescriptor(schema); } catch (RuntimeException e) { if (e.getMessage().startsWith("Reserved field name")) renameAndRetry(); else throw e; }

When it happens

Trigger: Defining a BigQuery sink schema that includes a field whose name collides with reserved CDC columns (record_name, record_index, changed_fields / the entries in StorageApiDirtyDataHoldingRecord / StorageApiCDC.COLUMNS) while using Storage API writes with CDC enabled.

Common situations: Ingesting upstream data that already has a column literally named like the CDC metadata column; adding a column called e.g. 'record_name' after CDC output exists in the same table.

Understand the failure class

Background: "invalid id" errors: invalid identifier format — why libraries reject IDs before lookup, and how to fix them — this error's family across 37 libraries.

Related errors

• A function must be provided to convert the input type into…
• BigQuery %1$s not found for table "%2$s" . Please create…
• BigQuery data contained value
• BigQuery temp location expected a valid 'gs://' path, but…
• BigQuery temp location expected a valid 'gs://' path, but…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5ca7e90897098355. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:1324

      fieldDescriptorBuilder = FieldDescriptorProto.newBuilder();
      fieldDescriptorBuilder = fieldDescriptorBuilder.setName(StorageApiCDC.CHANGE_SQN_COLUMN);
      fieldDescriptorBuilder = fieldDescriptorBuilder.setNumber(i++);
      fieldDescriptorBuilder =
          fieldDescriptorBuilder.setType(FieldDescriptorProto.Type.TYPE_STRING);
      fieldDescriptorBuilder = fieldDescriptorBuilder.setLabel(Label.LABEL_OPTIONAL);
      descriptorBuilder.addField(fieldDescriptorBuilder.build());
    }
    return descriptorBuilder.build();
  }

  private static void fieldDescriptorFromTableField(
      TableFieldSchema fieldSchema,
      int fieldNumber,
      DescriptorProto.Builder descriptorBuilder,
      boolean respectRequired) {
    if (StorageApiCDC.COLUMNS.contains(fieldSchema.getName())) {
      throw new RuntimeException(
          "Reserved field name " + fieldSchema.getName() + " in user schema.");
    }
    FieldDescriptorProto.Builder fieldDescriptorBuilder = FieldDescriptorProto.newBuilder();
    final String fieldName = fieldSchema.getName().toLowerCase();
    fieldDescriptorBuilder = fieldDescriptorBuilder.setName(fieldName);
    fieldDescriptorBuilder = fieldDescriptorBuilder.setNumber(fieldNumber);
    if (!BigQuerySchemaUtil.isProtoCompatible(fieldName)) {
      fieldDescriptorBuilder =
          fieldDescriptorBuilder.setName(
              BigQuerySchemaUtil.generatePlaceholderFieldName(fieldName));

      Message.Builder fieldOptionBuilder = DescriptorProtos.FieldOptions.newBuilder();
      fieldOptionBuilder =
          fieldOptionBuilder.setField(AnnotationsProto.columnName.getDescriptor(), fieldName);
      fieldDescriptorBuilder =
          fieldDescriptorBuilder.setOptions(
              (DescriptorProtos.FieldOptions) fieldOptionBuilder.build());
    }

View on GitHub (pinned to 12126d8942)