apache/iceberg · error · IllegalArgumentException

Invalid projection for field ${field.name()}: ${e.getMessage

Error message

Invalid projection for field ${field.name()}: ${e.getMessage()}

What it means

PruneColumnsWithoutReordering.field validates column projections against the table schema during scan planning; when the projection function throws IllegalArgumentException (requested struct/dataType does not match the schema field), it rethrows enriched with the offending field name. The original message is preserved as the cause.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java:138

      if (filterRefs.contains(field.fieldId())) {
        return field.type();
      }
      return null;
    }

    int fieldIndex = requestedStruct.fieldIndex(field.name());
    StructField requestedField = requestedStruct.fields()[fieldIndex];

    Preconditions.checkArgument(
        requestedField.nullable() || field.isRequired(),
        "Cannot project an optional field as non-null: %s",
        field.name());

    this.current = requestedField.dataType();
    try {
      return fieldResult.get();
    } catch (IllegalArgumentException e) {
      throw new IllegalArgumentException(
          "Invalid projection for field " + field.name() + ": " + e.getMessage(), e);
    } finally {
      this.current = requestedStruct;
    }
  }

  @Override
  public Type list(Types.ListType list, Supplier<Type> elementResult) {
    Preconditions.checkArgument(current instanceof ArrayType, "Not an array: %s", current);
    ArrayType requestedArray = (ArrayType) current;

    Preconditions.checkArgument(
        requestedArray.containsNull() || !list.isElementOptional(),
        "Cannot project an array of optional elements as required elements: %s",
        requestedArray);

    this.current = requestedArray.elementType();
    try {

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Refresh the table metadata and rebuild the query: spark.catalog().refreshTable(ident), then re-read the table.
  2. Inspect the chained cause in 'Invalid projection for field X: <cause>' and align the requested field name/type with the current schema.
  3. Clear stale Spark SQL caches referencing the old schema.
  4. If a custom reader/projection is involved, ensure the projected type exactly matches field.type().

Example fix

// before: stale cached DataFrame referencing an evolved schema
Dataset<Row> stale = spark.read().format("iceberg").load("db.t").cache();
// after
spark.catalog().refreshTable("db.t");
Dataset<Row> fresh = spark.read().format("iceberg").load("db.t");
Defensive patterns

Strategy: try-catch

Validate before calling

Types.NestedField field = schema.findField(columnName);
if (field == null) {
  // refresh metadata / rebuild the plan before projecting
}

Try / catch

try {
  df.select("col").collect();
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("Invalid projection for field")) {
    spark.catalog().refreshTable(tableIdent); // reload schema, retry once
  }
}

Prevention

When it happens

Trigger: A Spark scan whose requested projection type diverges from the Iceberg schema — e.g. pruning to nested columns after schema evolution renamed/dropped/retyped a field, or a reader requesting a projection that the field cannot satisfy.

Common situations: Schema evolution (column drop/rename/type change) with stale cached query plans or cached DataFrames; nested struct pruning where requested and actual struct types differ.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/1238ea016dfb476b. Report an issue: GitHub.