What causes "Cannot project an optional field as non-null: %s" in apache/iceberg?

Make the requested field nullable in the Spark schema

apache/iceberg · error · IllegalArgumentException

Cannot project an optional field as non-null: %s

Error message

Cannot project an optional field as non-null: %s

What it means

Thrown when a requested projection marks a field as non-nullable while the Iceberg field is optional (nullable) and the projected result is not required. Iceberg fields declared optional cannot be projected as non-null because values may legitimately be null.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java:128

  public Type field(Types.NestedField field, Supplier<Type> fieldResult) {
    Preconditions.checkArgument(current instanceof StructType, "Not a struct: %s", current);
    StructType requestedStruct = (StructType) current;

    // fields are resolved by name because Spark only sees the current table schema.
    if (requestedStruct.getFieldIndex(field.name()).isEmpty()) {
      // make sure that filter fields are projected even if they aren't in the requested schema.
      if (filterRefs.contains(field.fieldId())) {
        return field.type();
      }
      return null;
    }

    int fieldIndex = requestedStruct.fieldIndex(field.name());
    StructField requestedField = requestedStruct.fields()[fieldIndex];

    Preconditions.checkArgument(
        requestedField.nullable() || field.isRequired(),
        "Cannot project an optional field as non-null: %s",
        field.name());

    this.current = requestedField.dataType();
    try {
      return fieldResult.get();
    } catch (IllegalArgumentException e) {
      throw new IllegalArgumentException(
          "Invalid projection for field " + field.name() + ": " + e.getMessage(), e);
    } finally {
      this.current = requestedStruct;
    }
  }

  @Override
  public Type list(Types.ListType list, Supplier<Type> elementResult) {
    Preconditions.checkArgument(current instanceof ArrayType, "Not an array: %s", current);
    ArrayType requestedArray = (ArrayType) current;

View on GitHub (pinned to 86d9c8fc54)

Solutions

Make the requested field nullable in the Spark schema
Verify the table schema: if the column should be required, ensure the Iceberg field is required
Rebuild the requested schema from the table schema via SparkSchemaUtil.convert(table.schema())

Example fix

// before
StructField f = StructField.of("id", LongType, false);
// after
StructField f = StructField.of("id", LongType, true); // table column is optional

Defensive patterns

Strategy: validation

Validate before calling

StructField f = requested.apply("id"); if (!f.nullable() && table.schema().findField("id").isOptional()) { throw new IllegalStateException("id is optional in the table; request it as nullable"); }

Type guard

boolean nullabilityOk(StructField f, Types.NestedField icebergField) { return f.nullable() || icebergField.isRequired(); }

Try / catch

try { df = spark.read().schema(requestedSchema).load(...); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith("Cannot project an optional field as non-null")) { df = spark.read().schema(SparkSchemaUtil.convert(table.schema())).load(...); } else throw e; }

Prevention

Always mark columns nullable in read schemas unless the table schema says required
Regenerate schemas after schema evolution instead of caching old ones

When it happens

Trigger: Calling PruneColumnsWithoutReordering.field(field) where requestedField.nullable() is false, field.isRequired() is false, i.e. the requested Spark schema declares a NOT NULL field that the Iceberg table defines as optional.

Common situations: Hand-built Spark read schemas or views that incorrectly mark columns non-nullable; schema drift after the table evolved a column from required to optional; CTE/UDF outputs typed as non-null.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/f60ee1600e40d338. Report an issue: GitHub.