apache/iceberg · error · IllegalArgumentException

Incompatible change: cannot add required column

Error message

Incompatible change: cannot add required column: %s

What it means

Thrown when a Spark ALTER TABLE ADD COLUMN attempts to add a column declared NOT NULL (non-nullable). Iceberg supports required columns, but Spark DDL via Spark3Util forbids adding required columns to an existing table because existing rows would violate the requirement.

Solutions

1. Drop NOT NULL from the ADD COLUMN statement and use updateColumnNullability/set as required after backfill
2. Add the column as nullable, backfill, then enforce requiredness via UpdateSchema if the catalog supports it

Example fix

-- before
ALTER TABLE t ADD COLUMN c INT NOT NULL;
-- after
ALTER TABLE t ADD COLUMN c INT;

TableChange.AddColumn add = ...; if (!add.isNullable()) { throw new IllegalArgumentException("Add column must be nullable: " + String.join(".", add.fieldNames())); }

boolean canAdd(TableChange.AddColumn a) { return a.isNullable(); }

try { spark.sql("ALTER TABLE t ADD COLUMN c INT NOT NULL"); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith("Incompatible change: cannot add required column")) { spark.sql("ALTER TABLE t ADD COLUMN c INT"); } else throw e; }

When it happens

Trigger: ALTER TABLE ... ADD COLUMN c INT NOT NULL on an existing Iceberg table; AddColumn change with isNullable()==false.

Common situations: Porting DDL scripts written for non-Iceberg engines; backfill workflows that declare NOT NULL upfront.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Cannot add column since setting default values in Spark is…
• Cannot add column since setting default values in Spark is…
• Cannot convert unsupported type to Spark
• Cannot drop identifier fields in non-Iceberg table: $table
• Cannot project an optional field as non-null

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d5c132a8da6f1f0c. Report an issue: GitHub.

Appendix: source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java:240

    if (update.position() instanceof TableChange.After) {
      TableChange.After after = (TableChange.After) update.position();
      String referenceField = peerName(update.fieldNames(), after.column());
      pendingUpdate.moveAfter(DOT.join(update.fieldNames()), referenceField);

    } else if (update.position() instanceof TableChange.First) {
      pendingUpdate.moveFirst(DOT.join(update.fieldNames()));

    } else {
      throw new IllegalArgumentException("Unknown position for reorder: " + update.position());
    }
  }

  private static void apply(UpdateSchema pendingUpdate, TableChange.AddColumn add) {
    Preconditions.checkArgument(
        add.isNullable(),
        "Incompatible change: cannot add required column: %s",
        leafName(add.fieldNames()));
    if (add.defaultValue() != null) {
      throw new UnsupportedOperationException(
          String.format(
              "Cannot add column %s since setting default values in Spark is currently unsupported",
              leafName(add.fieldNames())));
    }

    Type type = SparkSchemaUtil.convert(add.dataType());
    pendingUpdate.addColumn(
        parentName(add.fieldNames()), leafName(add.fieldNames()), type, add.comment());

    if (add.position() instanceof TableChange.After) {
      TableChange.After after = (TableChange.After) add.position();
      String referenceField = peerName(add.fieldNames(), after.column());
      pendingUpdate.moveAfter(DOT.join(add.fieldNames()), referenceField);

    } else if (add.position() instanceof TableChange.First) {
      pendingUpdate.moveFirst(DOT.join(add.fieldNames()));

View on GitHub (pinned to 86d9c8fc54)