apache/beam · error · IllegalArgumentException

the following options are currently only available when read

Error message

the following options are currently only available when reading with Managed.ICEBERG_CDC: {invalidOptions}

What it means

IcebergScanConfig.validate enforces that options exclusive to CDC streaming reads (streaming_strategy/start_strategy like starting_strategy, and metadata_columns) are only supplied when reading via Managed.ICEBERG_CDC. If any such option is set on a non-CDC scan configuration, IllegalArgumentException is thrown listing the invalid options.

Source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergScanConfig.java:434

        invalidOptions.add("from_timestamp");
      }
      if (getToTimestamp() != null) {
        invalidOptions.add("to_timestamp");
      }
      if (getFromSnapshotInclusive() != null) {
        invalidOptions.add("from_snapshot");
      }
      if (getToSnapshot() != null) {
        invalidOptions.add("to_snapshot");
      }
      if (getStartingStrategy() != null) {
        invalidOptions.add("starting_strategy");
      }
      if (!getMetadataColumns().isEmpty()) {
        invalidOptions.add("metadata_columns");
      }
      if (!invalidOptions.isEmpty()) {
        throw new IllegalArgumentException(
            error(
                "the following options are currently only available when "
                    + "reading with Managed.ICEBERG_CDC: "
                    + invalidOptions));
      }
    } else {
      Set<Integer> primaryKeyIds = new HashSet<>(table.schema().identifierFieldIds());
      checkState(
          !primaryKeyIds.isEmpty(),
          "Cannot read CDC records as the table schema does not specified any primary key fields.");
      Set<Integer> projectedFieldIds = TypeUtil.getProjectedIds(getProjectedSchema());
      primaryKeyIds.removeAll(projectedFieldIds);
      checkArgument(
          primaryKeyIds.isEmpty(),
          "When reading CDC records, the projected schema must not drop primary key fields. "
              + "The specified configuration drops the following PK fields: %s",
          primaryKeyIds);
      validateMetadataColumns(table);

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove starting_strategy and metadata_columns from the configuration when not using Managed.ICEBERG_CDC.
2. Switch the read to Managed.ICEBERG_CDC if you actually need these options.
3. Build separate configuration objects for CDC and non-CDC reads instead of sharing one.
4. Conditionally set CDC-only options in pipeline construction code.

Example fix

// before
IcebergScanConfig config = IcebergScanConfig.builder()
    .setStartingStrategy("INCLUSIVE") // CDC-only
    .build();

// after
IcebergScanConfig config = IcebergScanConfig.builder().build(); // or use Managed.ICEBERG_CDC

if (!isCdcRead && (config.getStartingStrategy() != null || !config.getMetadataColumns().isEmpty())) {
  throw new IllegalArgumentException("starting_strategy/metadata_columns require Managed.ICEBERG_CDC");
}

try {
  scanConfig.validate();
} catch (IllegalArgumentException e) {
  LOG.error("CDC-only options in non-CDC scan: {}", e.getMessage());
}

When it happens

Trigger: Calling validate() (from expand or the thrown/keepException/dropException/... option paths) with starting_strategy or metadata_columns set while the scan is not a Managed.ICEBERG_CDC read, at IcebergScanConfig.java:434.

Common situations: Reusing one options object for both batch and CDC reads; leftover CDC options in a batch pipeline template; switching a pipeline from CDC to batch without clearing options.

Understand the failure class

Background: Conflicting config options: "cannot be used together" — configuration validation errors across open-source libraries — this error's family across 162 libraries.

Related errors

• Invalid starting strategy. Valid values are: {values}
• watermark_column_time_unit '{watermarkColumnTimeUnit}' is in
• Unexpected logical type: {}
• Unexpected Beam type: {}
• Unsupported CDC ValueKind: {}

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/d2001994f2f04917. Report an issue: GitHub.