apache/iceberg · error · java.lang.IllegalArgumentException

Invalid projection for field <field.name()>: <e.getMessage()

Error message

Invalid projection for field <field.name()>: <e.getMessage()>

What it means

PruneColumnsWithoutReordering.field projects a requested field type onto the current struct type during column pruning. If computing the projection throws IllegalArgumentException, it is re-wrapped with the field name: 'Invalid projection for field <name>: <msg>', giving context about which field failed.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java:138

      if (filterRefs.contains(field.fieldId())) {
        return field.type();
      }
      return null;
    }

    int fieldIndex = requestedStruct.fieldIndex(field.name());
    StructField requestedField = requestedStruct.fields()[fieldIndex];

    Preconditions.checkArgument(
        requestedField.nullable() || field.isRequired(),
        "Cannot project an optional field as non-null: %s",
        field.name());

    this.current = requestedField.dataType();
    try {
      return fieldResult.get();
    } catch (IllegalArgumentException e) {
      throw new IllegalArgumentException(
          "Invalid projection for field " + field.name() + ": " + e.getMessage(), e);
    } finally {
      this.current = requestedStruct;
    }
  }

  @Override
  public Type list(Types.ListType list, Supplier<Type> elementResult) {
    Preconditions.checkArgument(current instanceof ArrayType, "Not an array: %s", current);
    ArrayType requestedArray = (ArrayType) current;

    Preconditions.checkArgument(
        requestedArray.containsNull() || !list.isElementOptional(),
        "Cannot project an array of optional elements as required elements: %s",
        requestedArray);

    this.current = requestedArray.elementType();
    try {

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Read the inner message for the actual type mismatch and align the requested schema with the table's current schema
  2. Refresh cached tables/dataframes so plans use the latest table schema
  3. Re-check schema evolution changes that altered this field's type
  4. Report upstream if the mismatch comes from Iceberg's own pruning with identical schemas

Example fix

// before
spark.table("t").select("evolved_struct.newly_renamed_field")
// after
spark.table("t")            // refresh plan against current schema
  .select("evolved_struct.newly_renamed_field")
Defensive patterns

Strategy: try-catch

Try / catch

try {
  df = spark.read().format("iceberg").load(path);
} catch (IllegalArgumentException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Invalid projection for field")) {
    log.error("Schema drift on field; refresh schema. {}", e.getMessage(), e);
    spark.catalog().refreshTable(path);
  } else throw e;
}

Prevention

When it happens

Trigger: Schema pruning encounters a requested type incompatible with the table's current type for a given field (e.g. mismatched struct/array/map shapes after schema evolution or incompatible requested schema).

Common situations: Evolving a column's type in the table after queries/plans were built against the old schema; requesting a projection schema that diverges from the table schema; partition-transform mismatches.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/81b870601cf5865b. Report an issue: GitHub.