{"record":{"id":"70dcbe75c2e90f42","repo":"apache/beam","slug":"adding-required-columns-is-not-yet-supported-encountered","errorCode":null,"errorMessage":"Adding required columns is not yet supported. Encountered required columns: {requiredColumns}","messagePattern":"Adding required columns is not yet supported\\. Encountered required columns: (.+?)","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergCatalogConfig.java","lineNumber":247,"sourceCode":"      ImmutableList.Builder<String> requiredColumns = ImmutableList.builder();\n\n      for (Schema.Field col : columnsToAdd) {\n        String name = col.getName();\n        Type type = IcebergUtils.beamFieldTypeToIcebergFieldType(col.getType(), 0).type;\n        String desc = col.getDescription();\n\n        if (col.getType().getNullable()) {\n          if (desc.isEmpty()) {\n            update.addColumn(name, type);\n          } else {\n            update.addColumn(name, type, desc);\n          }\n        } else {\n          requiredColumns.add(name);\n        }\n      }\n      if (!requiredColumns.build().isEmpty()) {\n        throw new UnsupportedOperationException(\n            \"Adding required columns is not yet supported. \"\n                + \"Encountered required columns: \"\n                + requiredColumns.build());\n      }\n\n      columnsToDrop.forEach(update::deleteColumn);\n\n      update.commit();\n    }\n\n    public void updatePartitionSpec(\n        List<String> partitionsToAdd, Collection<String> partitionsToDrop) {\n      if (partitionsToAdd.isEmpty() && partitionsToDrop.isEmpty()) {\n        return;\n      }\n      UpdatePartitionSpec update = table.updateSpec();\n\n      partitionsToDrop.stream().map(PartitionUtils::toIcebergTerm).forEach(update::removeField);","sourceCodeStart":229,"sourceCodeEnd":265,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergCatalogConfig.java#L229-L265","documentation":"IcebergCatalogConfig.updateSchema computes the difference between the incoming Beam schema and the existing Iceberg schema. New columns that are non-nullable (required) cannot be added to an Iceberg table with existing data, so the operation fails fast with UnsupportedOperationException listing the offending columns.","triggerScenarios":"Calling updateSchema where the new schema contains a required (non-nullable) field that does not exist in the current Iceberg schema; requiredColumns is non-empty at IcebergCatalogConfig.java:247.","commonSituations":"Evolving a pipeline schema by adding a new non-null field to the Beam PCollection; enforcing NOT NULL on a newly introduced column; schema drift between the writer code and the deployed Iceberg table.","solutions":["Make the new column nullable (Schema.FieldType with nullable=true) so it can be added to the existing table.","Add the column as optional first, backfill existing rows with a batch job, then enforce NOT NULL at the engine/query level if needed.","Create a new table with the required schema and migrate data if strict non-null columns are mandatory.","Update the existing Iceberg table schema manually to match before running the pipeline."],"exampleFix":"// before\nSchema.Field.of(\"user_id\", Schema.FieldType.INT64);\n\n// after\nSchema.Field.of(\"user_id\", Schema.FieldType.INT64.withNullable(true));","handlingStrategy":"validation","validationCode":"List<String> required = newSchema.getFields().stream()\n    .filter(f -> !f.getType().getNullable())\n    .map(Schema.Field::getName)\n    .collect(Collectors.toList());\n// verify none of these are new vs. the existing Iceberg schema before calling updateSchema","typeGuard":null,"tryCatchPattern":"try {\n  config.updateSchema(newSchema);\n} catch (UnsupportedOperationException e) {\n  LOG.error(\"New required columns cannot be added: {}\", e.getMessage());\n}","preventionTips":["Keep all newly added fields nullable.","Review schema diffs before evolving tables with existing data.","Never enforce NOT NULL on brand-new columns in Iceberg.","Version schema changes through migration scripts."],"tags":["java","iceberg","schema-evolution","unsupported-operation"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}