{"record":{"id":"d5c132a8da6f1f0c","repo":"apache/iceberg","slug":"incompatible-change-cannot-add-required-column","errorCode":null,"errorMessage":"Incompatible change: cannot add required column: %s","messagePattern":"Incompatible change: cannot add required column: (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java","lineNumber":240,"sourceCode":"\n    if (update.position() instanceof TableChange.After) {\n      TableChange.After after = (TableChange.After) update.position();\n      String referenceField = peerName(update.fieldNames(), after.column());\n      pendingUpdate.moveAfter(DOT.join(update.fieldNames()), referenceField);\n\n    } else if (update.position() instanceof TableChange.First) {\n      pendingUpdate.moveFirst(DOT.join(update.fieldNames()));\n\n    } else {\n      throw new IllegalArgumentException(\"Unknown position for reorder: \" + update.position());\n    }\n  }\n\n  private static void apply(UpdateSchema pendingUpdate, TableChange.AddColumn add) {\n    Preconditions.checkArgument(\n        add.isNullable(),\n        \"Incompatible change: cannot add required column: %s\",\n        leafName(add.fieldNames()));\n    if (add.defaultValue() != null) {\n      throw new UnsupportedOperationException(\n          String.format(\n              \"Cannot add column %s since setting default values in Spark is currently unsupported\",\n              leafName(add.fieldNames())));\n    }\n\n    Type type = SparkSchemaUtil.convert(add.dataType());\n    pendingUpdate.addColumn(\n        parentName(add.fieldNames()), leafName(add.fieldNames()), type, add.comment());\n\n    if (add.position() instanceof TableChange.After) {\n      TableChange.After after = (TableChange.After) add.position();\n      String referenceField = peerName(add.fieldNames(), after.column());\n      pendingUpdate.moveAfter(DOT.join(add.fieldNames()), referenceField);\n\n    } else if (add.position() instanceof TableChange.First) {\n      pendingUpdate.moveFirst(DOT.join(add.fieldNames()));","sourceCodeStart":222,"sourceCodeEnd":258,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java#L222-L258","documentation":"Thrown when a Spark ALTER TABLE ADD COLUMN attempts to add a column declared NOT NULL (non-nullable). Iceberg supports required columns, but Spark DDL via Spark3Util forbids adding required columns to an existing table because existing rows would violate the requirement.","triggerScenarios":"ALTER TABLE ... ADD COLUMN c INT NOT NULL on an existing Iceberg table; AddColumn change with isNullable()==false.","commonSituations":"Porting DDL scripts written for non-Iceberg engines; backfill workflows that declare NOT NULL upfront.","solutions":["Drop NOT NULL from the ADD COLUMN statement and use updateColumnNullability/set as required after backfill","Add the column as nullable, backfill, then enforce requiredness via UpdateSchema if the catalog supports it"],"exampleFix":"-- before\nALTER TABLE t ADD COLUMN c INT NOT NULL;\n-- after\nALTER TABLE t ADD COLUMN c INT;","handlingStrategy":"validation","validationCode":"TableChange.AddColumn add = ...; if (!add.isNullable()) { throw new IllegalArgumentException(\"Add column must be nullable: \" + String.join(\".\", add.fieldNames())); }","typeGuard":"boolean canAdd(TableChange.AddColumn a) { return a.isNullable(); }","tryCatchPattern":"try { spark.sql(\"ALTER TABLE t ADD COLUMN c INT NOT NULL\"); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith(\"Incompatible change: cannot add required column\")) { spark.sql(\"ALTER TABLE t ADD COLUMN c INT\"); } else throw e; }","preventionTips":["Never emit NOT NULL in ALTER TABLE ADD COLUMN against Iceberg tables via Spark","Add nullable first, then tighten requiredness through Iceberg APIs after backfill"],"tags":["spark","schema","add-column"],"backgroundTag":"schema-validation-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}