{"record":{"id":"1724f444a0606f2c","repo":"apache/iceberg","slug":"invalid-projection-for-field-field-name-e","errorCode":null,"errorMessage":"Invalid projection for field ${field.name()}: ${e.getMessage()}","messagePattern":"Invalid projection for field (.+?): (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java","lineNumber":135,"sourceCode":"      if (filterRefs.contains(field.fieldId())) {\n        return field.type();\n      }\n      return null;\n    }\n\n    int fieldIndex = requestedStruct.fieldIndex(field.name());\n    StructField requestedField = requestedStruct.fields()[fieldIndex];\n\n    Preconditions.checkArgument(\n        requestedField.nullable() || field.isRequired(),\n        \"Cannot project an optional field as non-null: %s\",\n        field.name());\n\n    this.current = requestedField.dataType();\n    try {\n      return fieldResult.get();\n    } catch (IllegalArgumentException e) {\n      throw new IllegalArgumentException(\n          \"Invalid projection for field \" + field.name() + \": \" + e.getMessage(), e);\n    } finally {\n      this.current = requestedStruct;\n    }\n  }\n\n  @Override\n  public Type list(Types.ListType list, Supplier<Type> elementResult) {\n    Preconditions.checkArgument(current instanceof ArrayType, \"Not an array: %s\", current);\n    ArrayType requestedArray = (ArrayType) current;\n\n    Preconditions.checkArgument(\n        requestedArray.containsNull() || !list.isElementOptional(),\n        \"Cannot project an array of optional elements as required elements: %s\",\n        requestedArray);\n\n    this.current = requestedArray.elementType();\n    try {","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java#L117-L153","documentation":"PruneColumnsWithoutReordering.field wraps IllegalArgumentException raised while building a pruned projection for a requested field, rethrowing as 'Invalid projection for field <name>: <msg>'. This happens during column pruning when the requested (projected) type of a field cannot be produced from the current struct — an internal consistency check on projection validity.","triggerScenarios":"A Spark scan/batch read whose required schema contains an invalid projection for a field (e.g. requesting a nested/parameterized type that doesn't match the source field type, or a prune result rejected by the type system), typically via SparkScan/SparkInputPartition column pruning on a struct/list/map column.","commonSituations":"Reading Iceberg tables with deep nested schemas and a Spark projection (df.select on nested columns) that mismatches the stored type; schema evolution changing a field type between metadata and cached projection; bugs in custom scan/pruning extensions.","solutions":["Check that the select/projection targets match the actual nested field types in the table schema (df.schema vs table.schema).","Refresh metadata / drop stale cached plans (spark.catalog.clearCache()) after schema evolution, then re-run the read.","Simplify the projection — read the parent struct column and prune downstream instead of a partial nested projection that fails validation.","If reproducible on a plain scan with an unmodified schema, report as an Iceberg bug with the schema and projection (this wrapper preserves the root cause message)."],"exampleFix":"// before\ndf.select(\"nested.a.b\", \"other\")\n// after\n// ensure 'nested.a' is actually a struct containing 'b' before projecting\ndf.select(\"nested.a\", \"other\").select(\"a.b\", \"other\");","handlingStrategy":"try-catch","validationCode":"// verify requested nested type matches schema before selecting\nTypes.NestedField f = table.schema().findField(path);\nif (f == null) throw new IllegalArgumentException(\"Unknown column: \" + path);","typeGuard":null,"tryCatchPattern":"try { df.select(\"nested.a.b\"); } catch (IllegalArgumentException e) { LOG.error(\"Invalid projection: {}\", e.getMessage()); df = df.select(\"nested.a\"); }","preventionTips":["Confirm nested field types against table.schema() before df.select on nested paths","Clear Spark caches after schema evolution to avoid stale projections","Isolate complex nested pruning; fall back to reading parent structs if pruning fails"],"tags":["spark","column-pruning","projection"],"backgroundTag":"invalid-argument-value","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}