{"record":{"id":"1238ea016dfb476b","repo":"apache/iceberg","slug":"invalid-projection-for-field-field-name-e-1238ea","errorCode":null,"errorMessage":"Invalid projection for field ${field.name()}: ${e.getMessage()}","messagePattern":"Invalid projection for field (.+?): (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java","lineNumber":138,"sourceCode":"      if (filterRefs.contains(field.fieldId())) {\n        return field.type();\n      }\n      return null;\n    }\n\n    int fieldIndex = requestedStruct.fieldIndex(field.name());\n    StructField requestedField = requestedStruct.fields()[fieldIndex];\n\n    Preconditions.checkArgument(\n        requestedField.nullable() || field.isRequired(),\n        \"Cannot project an optional field as non-null: %s\",\n        field.name());\n\n    this.current = requestedField.dataType();\n    try {\n      return fieldResult.get();\n    } catch (IllegalArgumentException e) {\n      throw new IllegalArgumentException(\n          \"Invalid projection for field \" + field.name() + \": \" + e.getMessage(), e);\n    } finally {\n      this.current = requestedStruct;\n    }\n  }\n\n  @Override\n  public Type list(Types.ListType list, Supplier<Type> elementResult) {\n    Preconditions.checkArgument(current instanceof ArrayType, \"Not an array: %s\", current);\n    ArrayType requestedArray = (ArrayType) current;\n\n    Preconditions.checkArgument(\n        requestedArray.containsNull() || !list.isElementOptional(),\n        \"Cannot project an array of optional elements as required elements: %s\",\n        requestedArray);\n\n    this.current = requestedArray.elementType();\n    try {","sourceCodeStart":120,"sourceCodeEnd":156,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java#L120-L156","documentation":"PruneColumnsWithoutReordering.field validates column projections against the table schema during scan planning; when the projection function throws IllegalArgumentException (requested struct/dataType does not match the schema field), it rethrows enriched with the offending field name. The original message is preserved as the cause.","triggerScenarios":"A Spark scan whose requested projection type diverges from the Iceberg schema — e.g. pruning to nested columns after schema evolution renamed/dropped/retyped a field, or a reader requesting a projection that the field cannot satisfy.","commonSituations":"Schema evolution (column drop/rename/type change) with stale cached query plans or cached DataFrames; nested struct pruning where requested and actual struct types differ.","solutions":["Refresh the table metadata and rebuild the query: spark.catalog().refreshTable(ident), then re-read the table.","Inspect the chained cause in 'Invalid projection for field X: <cause>' and align the requested field name/type with the current schema.","Clear stale Spark SQL caches referencing the old schema.","If a custom reader/projection is involved, ensure the projected type exactly matches field.type()."],"exampleFix":"// before: stale cached DataFrame referencing an evolved schema\nDataset<Row> stale = spark.read().format(\"iceberg\").load(\"db.t\").cache();\n// after\nspark.catalog().refreshTable(\"db.t\");\nDataset<Row> fresh = spark.read().format(\"iceberg\").load(\"db.t\");","handlingStrategy":"try-catch","validationCode":"Types.NestedField field = schema.findField(columnName);\nif (field == null) {\n  // refresh metadata / rebuild the plan before projecting\n}","typeGuard":null,"tryCatchPattern":"try {\n  df.select(\"col\").collect();\n} catch (IllegalArgumentException e) {\n  if (e.getMessage().startsWith(\"Invalid projection for field\")) {\n    spark.catalog().refreshTable(tableIdent); // reload schema, retry once\n  }\n}","preventionTips":["Refresh table metadata after any schema evolution before reusing cached DataFrames.","Avoid caching Iceberg DataFrames across schema-changing operations.","Validate projected field names/types against the current schema before scanning."],"tags":["spark","schema","projection"],"backgroundTag":"schema-validation-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}