apache/beam · error · UnsupportedOperationException

Adding required columns is not yet supported. Encountered…

Error message

Adding required columns is not yet supported. Encountered required columns: {requiredColumns}

What it means

IcebergCatalogConfig.updateSchema computes the difference between the incoming Beam schema and the existing Iceberg schema. New columns that are non-nullable (required) cannot be added to an Iceberg table with existing data, so the operation fails fast with UnsupportedOperationException listing the offending columns.

Solutions

  1. Make the new column nullable (Schema.FieldType with nullable=true) so it can be added to the existing table.
  2. Add the column as optional first, backfill existing rows with a batch job, then enforce NOT NULL at the engine/query level if needed.
  3. Create a new table with the required schema and migrate data if strict non-null columns are mandatory.
  4. Update the existing Iceberg table schema manually to match before running the pipeline.

Example fix

// before
Schema.Field.of("user_id", Schema.FieldType.INT64);

// after
Schema.Field.of("user_id", Schema.FieldType.INT64.withNullable(true));
Defensive patterns

Strategy: validation

Validate before calling

List<String> required = newSchema.getFields().stream()
    .filter(f -> !f.getType().getNullable())
    .map(Schema.Field::getName)
    .collect(Collectors.toList());
// verify none of these are new vs. the existing Iceberg schema before calling updateSchema

Try / catch

try {
  config.updateSchema(newSchema);
} catch (UnsupportedOperationException e) {
  LOG.error("New required columns cannot be added: {}", e.getMessage());
}

Prevention

When it happens

Trigger: Calling updateSchema where the new schema contains a required (non-nullable) field that does not exist in the current Iceberg schema; requiredColumns is non-empty at IcebergCatalogConfig.java:247.

Common situations: Evolving a pipeline schema by adding a new non-null field to the Beam PCollection; enforcing NOT NULL on a newly introduced column; schema drift between the writer code and the deployed Iceberg table.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/70dcbe75c2e90f42. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergCatalogConfig.java:247

      ImmutableList.Builder<String> requiredColumns = ImmutableList.builder();

      for (Schema.Field col : columnsToAdd) {
        String name = col.getName();
        Type type = IcebergUtils.beamFieldTypeToIcebergFieldType(col.getType(), 0).type;
        String desc = col.getDescription();

        if (col.getType().getNullable()) {
          if (desc.isEmpty()) {
            update.addColumn(name, type);
          } else {
            update.addColumn(name, type, desc);
          }
        } else {
          requiredColumns.add(name);
        }
      }
      if (!requiredColumns.build().isEmpty()) {
        throw new UnsupportedOperationException(
            "Adding required columns is not yet supported. "
                + "Encountered required columns: "
                + requiredColumns.build());
      }

      columnsToDrop.forEach(update::deleteColumn);

      update.commit();
    }

    public void updatePartitionSpec(
        List<String> partitionsToAdd, Collection<String> partitionsToDrop) {
      if (partitionsToAdd.isEmpty() && partitionsToDrop.isEmpty()) {
        return;
      }
      UpdatePartitionSpec update = table.updateSpec();

      partitionsToDrop.stream().map(PartitionUtils::toIcebergTerm).forEach(update::removeField);

View on GitHub (pinned to 12126d8942)