apache/beam · error · IllegalArgumentException

Unsupported CDC metadata column

Error message

Unsupported CDC metadata column: {}

What it means

IcebergCdcMetadataColumns.beamField maps a CDC metadata column name to its Beam Schema.Field type. Only COMMIT_SNAPSHOT_ID, COMMIT_SNAPSHOT_SEQUENCE_NUMBER, ROW_ID, and LAST_UPDATED_SEQUENCE_NUMBER are supported; any other name throws IllegalArgumentException. This fails fast when a caller requests a metadata column the CDC implementation does not expose.

Solutions

1. Use only the constants defined in IcebergCdcMetadataColumns (COMMIT_SNAPSHOT_ID, COMMIT_SNAPSHOT_SEQUENCE_NUMBER, ROW_ID, LAST_UPDATED_SEQUENCE_NUMBER)
2. Validate configured column names against the supported set before building the pipeline
3. Add a mapping in beamField if the column genuinely needs support
4. Check for typos/case mismatches in the column name

Example fix

// before
Schema.Field f = IcebergCdcMetadataColumns.beamField("last_updated_snapshot_id");
// after
String name = IcebergCdcMetadataColumns.COMMIT_SNAPSHOT_ID; // use a supported constant
Schema.Field f = IcebergCdcMetadataColumns.beamField(name);

Set<String> supported = Set.of(IcebergCdcMetadataColumns.COMMIT_SNAPSHOT_ID,
    IcebergCdcMetadataColumns.COMMIT_SNAPSHOT_SEQUENCE_NUMBER,
    IcebergCdcMetadataColumns.ROW_ID,
    IcebergCdcMetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER);
if (!supported.contains(columnName)) {
  throw new IllegalArgumentException("Unsupported CDC metadata column: " + columnName);
}

try {
  Schema.Field f = IcebergCdcMetadataColumns.beamField(name);
} catch (IllegalArgumentException e) {
  LOG.warn("Falling back: {}", e.getMessage());
  f = null; // skip optional metadata column
}

When it happens

Trigger: Calling IcebergCdcMetadataColumns.beamField(name) with a name string that is not one of the four supported metadata column constants — typically a typo, a renamed constant, or a custom column name passed to CDC column configuration.

Common situations: Typo in a configured CDC metadata column name; using a metadata column name from Iceberg's MetadataColumns (e.g. ROW_DATA_SEQUENCE_NUMBER) that Beam's CDC layer does not support; upgrading Iceberg and assuming new metadata columns are exposed.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• Equality field is not a top-level column of schema
• Expected at least one overlapping task in bidirectional list
• Field must not be null.
• Invalid starting strategy. Valid values are
• Position delete index cardinality exceeds Integer.MAX_VALUE

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/4b910820639aa4eb. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/cdc/IcebergCdcMetadataColumns.java:80

  public static boolean isSupportedColumn(String name) {
    return SUPPORTED_COLUMNS.contains(name);
  }

  public static boolean isRowMetadataColumn(String name) {
    return ROW_METADATA_COLUMNS.contains(name);
  }

  public static Schema.Field beamField(String name) {
    if (CHANGE_TYPE.equals(name)) {
      return Schema.Field.of(name, Schema.FieldType.STRING);
    }
    if (COMMIT_SNAPSHOT_ID.equals(name) || COMMIT_SNAPSHOT_SEQUENCE_NUMBER.equals(name)) {
      return Schema.Field.of(name, Schema.FieldType.INT64);
    }
    if (ROW_ID.equals(name) || LAST_UPDATED_SEQUENCE_NUMBER.equals(name)) {
      return Schema.Field.nullable(name, Schema.FieldType.INT64);
    }
    throw new IllegalArgumentException("Unsupported CDC metadata column: " + name);
  }

  /** Returns the Iceberg reader field for row-sourced metadata, or null for commit metadata. */
  public static Types.@Nullable NestedField icebergRowMetadataField(String name) {
    if (ROW_ID.equals(name)) {
      return MetadataColumns.ROW_ID;
    }
    if (LAST_UPDATED_SEQUENCE_NUMBER.equals(name)) {
      return MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER;
    }
    return null;
  }
}

View on GitHub (pinned to 12126d8942)