{"record":{"id":"516cad9df3aea96e","repo":"apache/iceberg","slug":"invalid-projection-for-field-s-s","errorCode":null,"errorMessage":"Invalid projection for field %s: %s","messagePattern":"Invalid projection for field (.+?): (.+?)","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java","lineNumber":135,"sourceCode":"      if (filterRefs.contains(field.fieldId())) {\n        return field.type();\n      }\n      return null;\n    }\n\n    int fieldIndex = requestedStruct.fieldIndex(field.name());\n    StructField requestedField = requestedStruct.fields()[fieldIndex];\n\n    Preconditions.checkArgument(\n        requestedField.nullable() || field.isRequired(),\n        \"Cannot project an optional field as non-null: %s\",\n        field.name());\n\n    this.current = requestedField.dataType();\n    try {\n      return fieldResult.get();\n    } catch (IllegalArgumentException e) {\n      throw new IllegalArgumentException(\n          \"Invalid projection for field \" + field.name() + \": \" + e.getMessage(), e);\n    } finally {\n      this.current = requestedStruct;\n    }\n  }\n\n  @Override\n  public Type list(Types.ListType list, Supplier<Type> elementResult) {\n    Preconditions.checkArgument(current instanceof ArrayType, \"Not an array: %s\", current);\n    ArrayType requestedArray = (ArrayType) current;\n\n    Preconditions.checkArgument(\n        requestedArray.containsNull() || !list.isElementOptional(),\n        \"Cannot project an array of optional elements as required elements: %s\",\n        requestedArray);\n\n    this.current = requestedArray.elementType();\n    try {","sourceCodeStart":117,"sourceCodeEnd":153,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java#L117-L153","documentation":"Thrown when pruning columns during a Spark scan produces a projection that is invalid for a given Iceberg field, wrapping the underlying IllegalArgumentException from the projection computation. Iceberg validates that the requested Spark projection can be represented against the Iceberg field before applying it.","triggerScenarios":"Calling PruneColumnsWithoutReordering.field(field) when the nested projection function throws IllegalArgumentException, e.g. projecting with an incompatible type or unsupported nested prune.","commonSituations":"Pushing Spark column pruning into Iceberg scans when schemas diverge between Spark and Iceberg (e.g. after schema evolution or incompatible casts in the requested schema).","solutions":["Inspect the wrapped cause message to find the actual invalid projection","Ensure the requested Spark schema matches the Iceberg table schema types","Update to a Spark/Iceberg version where the prune logic handles the type","Avoid pruning that changes nullability of nested fields"],"exampleFix":"// before\nStructType pruned = pruneColumns(schema.asStruct(), requested);\n// after\nStructType pruned = pruneColumns(schema.asStruct(), requested); // keep requested schema derived from table.schema() via SparkSchemaUtil.convert(table.schema())","handlingStrategy":"validation","validationCode":"if (!requested.schemaEquals(SparkSchemaUtil.convert(table.schema()))) { throw new IllegalArgumentException(\"Requested schema diverges from table schema; rebuild it from table.schema()\"); }","typeGuard":"boolean isCompatible(StructType requested, Schema table) { return requested.equals(SparkSchemaUtil.convert(table)); }","tryCatchPattern":"try { StructType pruned = pruneColumns(schema.asStruct(), requested); } catch (IllegalArgumentException e) { LOG.error(\"Projection failed: {}\", e.getMessage(), e); throw new AnalysisException(e.getMessage(), e); }","preventionTips":["Derive requested schemas from table.schema(), never hand-build","Keep Spark and Iceberg connector versions aligned"],"tags":["spark","schema","column-pruning"],"backgroundTag":"schema-validation-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}