apache/iceberg · error · UnsupportedOperationException

Cannot add column %s since setting default values in Spark i

Error message

Cannot add column %s since setting default values in Spark is currently unsupported

What it means

Iceberg's Spark DDL integration in this version cannot set column default values when adding columns. If a TableChange.AddColumn carries a non-null defaultValue, an UnsupportedOperationException naming the column is thrown.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java:238

      TableChange.After after = (TableChange.After) update.position();
      String referenceField = peerName(update.fieldNames(), after.column());
      pendingUpdate.moveAfter(DOT.join(update.fieldNames()), referenceField);

    } else if (update.position() instanceof TableChange.First) {
      pendingUpdate.moveFirst(DOT.join(update.fieldNames()));

    } else {
      throw new IllegalArgumentException("Unknown position for reorder: " + update.position());
    }
  }

  private static void apply(UpdateSchema pendingUpdate, TableChange.AddColumn add) {
    Preconditions.checkArgument(
        add.isNullable(),
        "Incompatible change: cannot add required column: %s",
        leafName(add.fieldNames()));
    if (add.defaultValue() != null) {
      throw new UnsupportedOperationException(
          String.format(
              "Cannot add column %s since setting default values in Spark is currently unsupported",
              leafName(add.fieldNames())));
    }

    Type type = SparkSchemaUtil.convert(add.dataType());
    pendingUpdate.addColumn(
        parentName(add.fieldNames()), leafName(add.fieldNames()), type, add.comment());

    if (add.position() instanceof TableChange.After) {
      TableChange.After after = (TableChange.After) add.position();
      String referenceField = peerName(add.fieldNames(), after.column());
      pendingUpdate.moveAfter(DOT.join(add.fieldNames()), referenceField);

    } else if (add.position() instanceof TableChange.First) {
      pendingUpdate.moveFirst(DOT.join(add.fieldNames()));

    } else {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Remove the DEFAULT clause and add the column without a default
2. Add the column, then backfill existing/default rows with UPDATE
3. Upgrade to an Iceberg version that supports column default values (schema spec 'defaults' support)

Example fix

// before
ALTER TABLE t ADD COLUMN c INT DEFAULT 42;
// after
ALTER TABLE t ADD COLUMN c INT;

boolean hasDefaults = changes.stream().filter(c -> c instanceof TableChange.AddColumn).map(c -> (TableChange.AddColumn) c).anyMatch(a -> a.defaultValue() != null);
if (hasDefaults) throw new IllegalArgumentException("column defaults unsupported by this Iceberg Spark version");

boolean addColumnHasDefault(TableChange c) { return c instanceof TableChange.AddColumn && ((TableChange.AddColumn) c).defaultValue() != null; }

try { Spark3Util.applySchemaChanges(table, changes); } catch (UnsupportedOperationException e) { if (e.getMessage().contains("setting default values")) { /* strip defaults and retry */ } else throw e; }

When it happens

Trigger: CREATE/ALTER TABLE adding a column with a DEFAULT clause (Spark 3.4+ DEFAULT syntax or SQL pipe syntax), producing an AddColumn change with a non-null default value.

Common situations: Migrating Delta/Hive DDL with DEFAULT column values to Iceberg on Spark 3.4/3.5; ORC/Parquet-generated schemas including defaults.

Related errors

• Creating table with computed columns is not supported yet.
• Creating table with watermark specs is not supported yet.
• Unsupported format in USING: ${provider}
• Transform is not supported: ${transform}
• Cannot convert unknown expression: ${expr}

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/157fecbac86841cd. Report an issue: GitHub.