apache/iceberg · error · IllegalArgumentException

Invalid projection for field ${field.name()}: ${e.getMessage

Error message

Invalid projection for field ${field.name()}: ${e.getMessage()}

What it means

PruneColumnsWithoutReordering.field wraps IllegalArgumentException raised while building a pruned projection for a requested field, rethrowing as 'Invalid projection for field <name>: <msg>'. This happens during column pruning when the requested (projected) type of a field cannot be produced from the current struct — an internal consistency check on projection validity.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java:135

      if (filterRefs.contains(field.fieldId())) {
        return field.type();
      }
      return null;
    }

    int fieldIndex = requestedStruct.fieldIndex(field.name());
    StructField requestedField = requestedStruct.fields()[fieldIndex];

    Preconditions.checkArgument(
        requestedField.nullable() || field.isRequired(),
        "Cannot project an optional field as non-null: %s",
        field.name());

    this.current = requestedField.dataType();
    try {
      return fieldResult.get();
    } catch (IllegalArgumentException e) {
      throw new IllegalArgumentException(
          "Invalid projection for field " + field.name() + ": " + e.getMessage(), e);
    } finally {
      this.current = requestedStruct;
    }
  }

  @Override
  public Type list(Types.ListType list, Supplier<Type> elementResult) {
    Preconditions.checkArgument(current instanceof ArrayType, "Not an array: %s", current);
    ArrayType requestedArray = (ArrayType) current;

    Preconditions.checkArgument(
        requestedArray.containsNull() || !list.isElementOptional(),
        "Cannot project an array of optional elements as required elements: %s",
        requestedArray);

    this.current = requestedArray.elementType();
    try {

View on GitHub (pinned to 86d9c8fc54)

Solutions

Check that the select/projection targets match the actual nested field types in the table schema (df.schema vs table.schema).
Refresh metadata / drop stale cached plans (spark.catalog.clearCache()) after schema evolution, then re-run the read.
Simplify the projection — read the parent struct column and prune downstream instead of a partial nested projection that fails validation.
If reproducible on a plain scan with an unmodified schema, report as an Iceberg bug with the schema and projection (this wrapper preserves the root cause message).

Example fix

// before
df.select("nested.a.b", "other")
// after
// ensure 'nested.a' is actually a struct containing 'b' before projecting
df.select("nested.a", "other").select("a.b", "other");

Defensive patterns

Strategy: try-catch

Validate before calling

// verify requested nested type matches schema before selecting
Types.NestedField f = table.schema().findField(path);
if (f == null) throw new IllegalArgumentException("Unknown column: " + path);

Try / catch

try { df.select("nested.a.b"); } catch (IllegalArgumentException e) { LOG.error("Invalid projection: {}", e.getMessage()); df = df.select("nested.a"); }

Prevention

Confirm nested field types against table.schema() before df.select on nested paths
Clear Spark caches after schema evolution to avoid stale projections
Isolate complex nested pruning; fall back to reading parent structs if pruning fails

When it happens

Trigger: A Spark scan/batch read whose required schema contains an invalid projection for a field (e.g. requesting a nested/parameterized type that doesn't match the source field type, or a prune result rejected by the type system), typically via SparkScan/SparkInputPartition column pruning on a struct/list/map column.

Common situations: Reading Iceberg tables with deep nested schemas and a Spark projection (df.select on nested columns) that mismatches the stored type; schema evolution changing a field type between metadata and cached projection; bugs in custom scan/pruning extensions.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/1724f444a0606f2c. Report an issue: GitHub.