apache/iceberg · error · IllegalArgumentException

Invalid projection for field %s: %s

Error message

Invalid projection for field %s: %s

What it means

Thrown when pruning columns during a Spark scan produces a projection that is invalid for a given Iceberg field, wrapping the underlying IllegalArgumentException from the projection computation. Iceberg validates that the requested Spark projection can be represented against the Iceberg field before applying it.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java:135

      if (filterRefs.contains(field.fieldId())) {
        return field.type();
      }
      return null;
    }

    int fieldIndex = requestedStruct.fieldIndex(field.name());
    StructField requestedField = requestedStruct.fields()[fieldIndex];

    Preconditions.checkArgument(
        requestedField.nullable() || field.isRequired(),
        "Cannot project an optional field as non-null: %s",
        field.name());

    this.current = requestedField.dataType();
    try {
      return fieldResult.get();
    } catch (IllegalArgumentException e) {
      throw new IllegalArgumentException(
          "Invalid projection for field " + field.name() + ": " + e.getMessage(), e);
    } finally {
      this.current = requestedStruct;
    }
  }

  @Override
  public Type list(Types.ListType list, Supplier<Type> elementResult) {
    Preconditions.checkArgument(current instanceof ArrayType, "Not an array: %s", current);
    ArrayType requestedArray = (ArrayType) current;

    Preconditions.checkArgument(
        requestedArray.containsNull() || !list.isElementOptional(),
        "Cannot project an array of optional elements as required elements: %s",
        requestedArray);

    this.current = requestedArray.elementType();
    try {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Inspect the wrapped cause message to find the actual invalid projection
2. Ensure the requested Spark schema matches the Iceberg table schema types
3. Update to a Spark/Iceberg version where the prune logic handles the type
4. Avoid pruning that changes nullability of nested fields

Example fix

// before
StructType pruned = pruneColumns(schema.asStruct(), requested);
// after
StructType pruned = pruneColumns(schema.asStruct(), requested); // keep requested schema derived from table.schema() via SparkSchemaUtil.convert(table.schema())

if (!requested.schemaEquals(SparkSchemaUtil.convert(table.schema()))) { throw new IllegalArgumentException("Requested schema diverges from table schema; rebuild it from table.schema()"); }

boolean isCompatible(StructType requested, Schema table) { return requested.equals(SparkSchemaUtil.convert(table)); }

try { StructType pruned = pruneColumns(schema.asStruct(), requested); } catch (IllegalArgumentException e) { LOG.error("Projection failed: {}", e.getMessage(), e); throw new AnalysisException(e.getMessage(), e); }

When it happens

Trigger: Calling PruneColumnsWithoutReordering.field(field) when the nested projection function throws IllegalArgumentException, e.g. projecting with an incompatible type or unsupported nested prune.

Common situations: Pushing Spark column pruning into Iceberg scans when schemas diverge between Spark and Iceberg (e.g. after schema evolution or incompatible casts in the requested schema).

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Invalid projection for field ${field.name()}: ${e.getMessage
• Spark does not support time fields
• Unsupported type:
• Encountered an unsupported ORC type during a write from Spar
• Cannot project an optional field as non-null: %s

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/516cad9df3aea96e. Report an issue: GitHub.