{"record":{"id":"f60ee1600e40d338","repo":"apache/iceberg","slug":"cannot-project-an-optional-field-as-non-null-s","errorCode":null,"errorMessage":"Cannot project an optional field as non-null: %s","messagePattern":"Cannot project an optional field as non-null: (.+?)","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java","lineNumber":128,"sourceCode":"  public Type field(Types.NestedField field, Supplier<Type> fieldResult) {\n    Preconditions.checkArgument(current instanceof StructType, \"Not a struct: %s\", current);\n    StructType requestedStruct = (StructType) current;\n\n    // fields are resolved by name because Spark only sees the current table schema.\n    if (requestedStruct.getFieldIndex(field.name()).isEmpty()) {\n      // make sure that filter fields are projected even if they aren't in the requested schema.\n      if (filterRefs.contains(field.fieldId())) {\n        return field.type();\n      }\n      return null;\n    }\n\n    int fieldIndex = requestedStruct.fieldIndex(field.name());\n    StructField requestedField = requestedStruct.fields()[fieldIndex];\n\n    Preconditions.checkArgument(\n        requestedField.nullable() || field.isRequired(),\n        \"Cannot project an optional field as non-null: %s\",\n        field.name());\n\n    this.current = requestedField.dataType();\n    try {\n      return fieldResult.get();\n    } catch (IllegalArgumentException e) {\n      throw new IllegalArgumentException(\n          \"Invalid projection for field \" + field.name() + \": \" + e.getMessage(), e);\n    } finally {\n      this.current = requestedStruct;\n    }\n  }\n\n  @Override\n  public Type list(Types.ListType list, Supplier<Type> elementResult) {\n    Preconditions.checkArgument(current instanceof ArrayType, \"Not an array: %s\", current);\n    ArrayType requestedArray = (ArrayType) current;\n","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/PruneColumnsWithoutReordering.java#L110-L146","documentation":"Thrown when a requested projection marks a field as non-nullable while the Iceberg field is optional (nullable) and the projected result is not required. Iceberg fields declared optional cannot be projected as non-null because values may legitimately be null.","triggerScenarios":"Calling PruneColumnsWithoutReordering.field(field) where requestedField.nullable() is false, field.isRequired() is false, i.e. the requested Spark schema declares a NOT NULL field that the Iceberg table defines as optional.","commonSituations":"Hand-built Spark read schemas or views that incorrectly mark columns non-nullable; schema drift after the table evolved a column from required to optional; CTE/UDF outputs typed as non-null.","solutions":["Make the requested field nullable in the Spark schema","Verify the table schema: if the column should be required, ensure the Iceberg field is required","Rebuild the requested schema from the table schema via SparkSchemaUtil.convert(table.schema())"],"exampleFix":"// before\nStructField f = StructField.of(\"id\", LongType, false);\n// after\nStructField f = StructField.of(\"id\", LongType, true); // table column is optional","handlingStrategy":"validation","validationCode":"StructField f = requested.apply(\"id\"); if (!f.nullable() && table.schema().findField(\"id\").isOptional()) { throw new IllegalStateException(\"id is optional in the table; request it as nullable\"); }","typeGuard":"boolean nullabilityOk(StructField f, Types.NestedField icebergField) { return f.nullable() || icebergField.isRequired(); }","tryCatchPattern":"try { df = spark.read().schema(requestedSchema).load(...); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith(\"Cannot project an optional field as non-null\")) { df = spark.read().schema(SparkSchemaUtil.convert(table.schema())).load(...); } else throw e; }","preventionTips":["Always mark columns nullable in read schemas unless the table schema says required","Regenerate schemas after schema evolution instead of caching old ones"],"tags":["spark","schema","nullability"],"backgroundTag":"type-mismatch","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}